[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:11 dist-test-slave-9bxp krb5kdc[13697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:11 dist-test-slave-9bxp krb5kdc[13697](info): set up 2 sockets
Apr 30 08:00:11 dist-test-slave-9bxp krb5kdc[13697](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260430 08:00:13.338455 13690 mini_kdc.cc:121] Time spent starting KDC: real 2.004s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:00:13 dist-test-slave-9bxp krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:13.480201 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:34499
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:34499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:13.586181 13713 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:13.586436 13713 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:13.586527 13713 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:13.589824 13713 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:13.589921 13713 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:13.589974 13713 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:13.590030 13713 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:13.590075 13713 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:13.594399 13713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:34499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:34499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:13.595510 13713 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:13.596369 13713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:13.602186 13719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:13.602206 13721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:13.602207 13718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:13.602269 13713 server_base.cc:1061] running on GCE node
I20260430 08:00:13.602943 13713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:13.603812 13713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:13.604977 13713 hybrid_clock.cc:648] HybridClock initialized: now 1777536013604952 us; error 34 us; skew 500 ppm
Apr 30 08:00:13 dist-test-slave-9bxp krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536013, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:13.608170 13713 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:13.609349 13713 webserver.cc:492] Webserver started at http://127.13.94.190:38749/ using document root <none> and password file <none>
I20260430 08:00:13.609952 13713 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:13.610009 13713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:13.610231 13713 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:13.611922 13713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "892ef7d7699b41ee828a42de04d40313"
format_stamp: "Formatted at 2026-04-30 08:00:13 on dist-test-slave-9bxp"
server_key: "a2d0831a15b04a475e58878784fa7671"
server_key_iv: "0e7ac3a31c3e345b85d0a64e7d961aa2"
server_key_version: "encryptionkey@0"
I20260430 08:00:13.612413 13713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "892ef7d7699b41ee828a42de04d40313"
format_stamp: "Formatted at 2026-04-30 08:00:13 on dist-test-slave-9bxp"
server_key: "a2d0831a15b04a475e58878784fa7671"
server_key_iv: "0e7ac3a31c3e345b85d0a64e7d961aa2"
server_key_version: "encryptionkey@0"
I20260430 08:00:13.615882 13713 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260430 08:00:13.618023 13728 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:13.619001 13713 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260430 08:00:13.619149 13713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "892ef7d7699b41ee828a42de04d40313"
format_stamp: "Formatted at 2026-04-30 08:00:13 on dist-test-slave-9bxp"
server_key: "a2d0831a15b04a475e58878784fa7671"
server_key_iv: "0e7ac3a31c3e345b85d0a64e7d961aa2"
server_key_version: "encryptionkey@0"
I20260430 08:00:13.619252 13713 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:13.641335 13713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:13.644006 13713 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:13.644200 13713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:13.651785 13713 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:34499
I20260430 08:00:13.651808 13780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:34499 every 8 connection(s)
I20260430 08:00:13.652848 13713 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:13.655512 13781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:13.656423 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 13713
I20260430 08:00:13.656579 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:13.657559 13690 external_mini_cluster.cc:1468] Setting key 88faa9303f9a606d7472adadaed05c5b
I20260430 08:00:13.661722 13781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313: Bootstrap starting.
I20260430 08:00:13.663789 13781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:13.664479 13781 log.cc:826] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:13.666370 13781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313: No bootstrap required, opened a new log
I20260430 08:00:13.669066 13781 raft_consensus.cc:359] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ef7d7699b41ee828a42de04d40313" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 34499 } }
I20260430 08:00:13.669275 13781 raft_consensus.cc:385] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:13.669354 13781 raft_consensus.cc:740] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892ef7d7699b41ee828a42de04d40313, State: Initialized, Role: FOLLOWER
Apr 30 08:00:13 dist-test-slave-9bxp krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:13.669822 13781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ef7d7699b41ee828a42de04d40313" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 34499 } }
I20260430 08:00:13.669982 13781 raft_consensus.cc:399] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:13.670046 13781 raft_consensus.cc:493] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:13.670135 13781 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:13.671268 13781 raft_consensus.cc:515] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ef7d7699b41ee828a42de04d40313" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 34499 } }
I20260430 08:00:13.671684 13781 leader_election.cc:304] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 892ef7d7699b41ee828a42de04d40313; no voters:
I20260430 08:00:13.672079 13781 leader_election.cc:290] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:13.672189 13786 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:13.672400 13786 raft_consensus.cc:697] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [term 1 LEADER]: Becoming Leader. State: Replica: 892ef7d7699b41ee828a42de04d40313, State: Running, Role: LEADER
I20260430 08:00:13.672739 13786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ef7d7699b41ee828a42de04d40313" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 34499 } }
I20260430 08:00:13.673498 13781 sys_catalog.cc:565] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:13.673890 13786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892ef7d7699b41ee828a42de04d40313. Latest consensus state: current_term: 1 leader_uuid: "892ef7d7699b41ee828a42de04d40313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ef7d7699b41ee828a42de04d40313" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 34499 } } }
I20260430 08:00:13.674017 13786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:13.675618 13787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "892ef7d7699b41ee828a42de04d40313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ef7d7699b41ee828a42de04d40313" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 34499 } } }
I20260430 08:00:13.675832 13787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:13.676645 13783 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:13.659734 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.13.94.190:34499)
0430 08:00:13.660117 (+ 383us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:13.660130 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:13.660681 (+ 551us) server_negotiation.cc:408] Connection header received
0430 08:00:13.662966 (+ 2285us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:13.662980 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:13.663237 (+ 257us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:13.663539 (+ 302us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:13.665507 (+ 1968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:13.666247 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:13.666978 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:13.667239 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:13.670659 (+ 3420us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:13.670685 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:13.670698 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:13.670735 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:13.673570 (+ 2835us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:13.674237 (+ 667us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:13.674243 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:13.674249 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:13.674348 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:13.674669 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:13.674676 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:13.674678 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:13.675041 (+ 363us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:13.675246 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:13.675912 (+ 666us) server_negotiation.cc:300] Negotiation successful
0430 08:00:13.676099 (+ 187us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
W20260430 08:00:13.678802 13802 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 08:00:13.678875 13802 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 08:00:13.678954 13790 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:13.679724 13790 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:13.684329 13790 catalog_manager.cc:1357] Generated new cluster ID: 8a7b59551f894b5bbdfe19cd8bd9e700
I20260430 08:00:13.684391 13790 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:13.702055 13790 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:13.703230 13790 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:13.714813 13790 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 892ef7d7699b41ee828a42de04d40313: Generated new TSK 0
I20260430 08:00:13.715517 13790 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:13.772451 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:34499
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:13.877290 13809 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:13.877502 13809 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:13.877560 13809 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:13.880894 13809 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:13.880957 13809 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:13.881036 13809 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:13.885190 13809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:34499
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:13.886175 13809 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:13.886963 13809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:13.893401 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:13.893391 13815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:13.893501 13817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:13.893558 13809 server_base.cc:1061] running on GCE node
I20260430 08:00:13.894091 13809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:13.894611 13809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:13.895764 13809 hybrid_clock.cc:648] HybridClock initialized: now 1777536013895737 us; error 38 us; skew 500 ppm
Apr 30 08:00:13 dist-test-slave-9bxp krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536013, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:13.898335 13809 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:00:13.899297 13809 webserver.cc:492] Webserver started at http://127.13.94.129:32797/ using document root <none> and password file <none>
I20260430 08:00:13.899808 13809 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:13.899854 13809 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:13.900010 13809 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:13.901693 13809 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "e798d89e008c4674bfea643e984d4ffb"
format_stamp: "Formatted at 2026-04-30 08:00:13 on dist-test-slave-9bxp"
server_key: "c98d5a312cea92720636f67a5764439c"
server_key_iv: "99f630566228b4731eababa3b8578a27"
server_key_version: "encryptionkey@0"
I20260430 08:00:13.902108 13809 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "e798d89e008c4674bfea643e984d4ffb"
format_stamp: "Formatted at 2026-04-30 08:00:13 on dist-test-slave-9bxp"
server_key: "c98d5a312cea92720636f67a5764439c"
server_key_iv: "99f630566228b4731eababa3b8578a27"
server_key_version: "encryptionkey@0"
I20260430 08:00:13.905267 13809 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 08:00:13.907382 13824 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:13.908407 13809 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:13.908525 13809 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "e798d89e008c4674bfea643e984d4ffb"
format_stamp: "Formatted at 2026-04-30 08:00:13 on dist-test-slave-9bxp"
server_key: "c98d5a312cea92720636f67a5764439c"
server_key_iv: "99f630566228b4731eababa3b8578a27"
server_key_version: "encryptionkey@0"
I20260430 08:00:13.908622 13809 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:13.923316 13809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:13.926285 13809 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:13.926491 13809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:13.927078 13809 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:13.927960 13809 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:13.928016 13809 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:13.928084 13809 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:13.928115 13809 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:13.937250 13809 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:38611
I20260430 08:00:13.937266 13937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:38611 every 8 connection(s)
I20260430 08:00:13.938235 13809 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
Apr 30 08:00:13 dist-test-slave-9bxp krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536013, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:13.948230 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 13809
I20260430 08:00:13.948343 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:13.948576 13690 external_mini_cluster.cc:1468] Setting key e3a7701b06c0b8582c1cdc507d4e69b6
I20260430 08:00:13.950631 13783 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:13.940007 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:60421 (local address 127.13.94.190:34499)
0430 08:00:13.940179 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:13.940184 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:13.940908 (+ 724us) server_negotiation.cc:408] Connection header received
0430 08:00:13.941728 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:13.941731 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:13.941784 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:13.941863 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:13.943389 (+ 1526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:13.944050 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:13.944775 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:13.944922 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:13.946956 (+ 2034us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:13.946975 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:13.946977 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:13.947004 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:13.948588 (+ 1584us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:13.949184 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:13.949188 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:13.949190 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:13.949243 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:13.949583 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:13.949586 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:13.949588 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:13.949746 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:13.949823 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:13.950346 (+ 523us) server_negotiation.cc:300] Negotiation successful
0430 08:00:13.950474 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260430 08:00:13.951248 13940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:13.940308 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:34499 (local address 127.13.94.129:60421)
0430 08:00:13.940777 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:13.940811 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:13.941533 (+ 722us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:13.942008 (+ 475us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:13.942016 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:13.942386 (+ 370us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:13.943240 (+ 854us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:13.943252 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:13.944233 (+ 981us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:13.944236 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:13.944624 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:13.944630 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:13.944862 (+ 232us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:13.945384 (+ 522us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:13.945402 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:13.946808 (+ 1406us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:13.948756 (+ 1948us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:13.948763 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:13.948779 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:13.949077 (+ 298us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:13.949341 (+ 264us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:13.949344 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:13.949345 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:13.949477 (+ 132us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:13.949850 (+ 373us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:13.949856 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:13.950095 (+ 239us) client_negotiation.cc:770] Sending connection context
0430 08:00:13.950296 (+ 201us) client_negotiation.cc:241] Negotiation successful
0430 08:00:13.950528 (+ 232us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
I20260430 08:00:13.952366 13938 heartbeater.cc:344] Connected to a master server at 127.13.94.190:34499
I20260430 08:00:13.952606 13938 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:13.953140 13938 heartbeater.cc:507] Master 127.13.94.190:34499 requested a full tablet report, sending...
I20260430 08:00:13.954581 13745 ts_manager.cc:194] Registered new tserver with Master: e798d89e008c4674bfea643e984d4ffb (127.13.94.129:38611)
I20260430 08:00:13.955765 13745 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:60421
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:13.999375 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:34499
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:14.100127 13945 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:14.100317 13945 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:14.100420 13945 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:14.103703 13945 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:14.103763 13945 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:14.103891 13945 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:14.108065 13945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:34499
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:14.109167 13945 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:14.110001 13945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:14.116344 13950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:14.116448 13951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:14.116530 13953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:14.116787 13945 server_base.cc:1061] running on GCE node
I20260430 08:00:14.117128 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:14.117641 13945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:14.118827 13945 hybrid_clock.cc:648] HybridClock initialized: now 1777536014118811 us; error 37 us; skew 500 ppm
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536014, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:14.121424 13945 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:00:14.122416 13945 webserver.cc:492] Webserver started at http://127.13.94.130:41971/ using document root <none> and password file <none>
I20260430 08:00:14.122948 13945 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:14.123025 13945 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:14.123227 13945 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:14.124992 13945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "e0d525d0162a4ae8bd19af88ed6b1637"
format_stamp: "Formatted at 2026-04-30 08:00:14 on dist-test-slave-9bxp"
server_key: "5a7c03b02a2a8df439fd20aae44af4b4"
server_key_iv: "7d6a3249db8259afb36672c751edab5d"
server_key_version: "encryptionkey@0"
I20260430 08:00:14.125478 13945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "e0d525d0162a4ae8bd19af88ed6b1637"
format_stamp: "Formatted at 2026-04-30 08:00:14 on dist-test-slave-9bxp"
server_key: "5a7c03b02a2a8df439fd20aae44af4b4"
server_key_iv: "7d6a3249db8259afb36672c751edab5d"
server_key_version: "encryptionkey@0"
I20260430 08:00:14.128850 13945 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:00:14.131001 13960 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:14.132140 13945 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:14.132284 13945 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "e0d525d0162a4ae8bd19af88ed6b1637"
format_stamp: "Formatted at 2026-04-30 08:00:14 on dist-test-slave-9bxp"
server_key: "5a7c03b02a2a8df439fd20aae44af4b4"
server_key_iv: "7d6a3249db8259afb36672c751edab5d"
server_key_version: "encryptionkey@0"
I20260430 08:00:14.132388 13945 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:14.142980 13945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:14.145705 13945 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:14.145900 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:14.146541 13945 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:14.147388 13945 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:14.147457 13945 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:14.147540 13945 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:14.147581 13945 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:14.156865 13945 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:35793
I20260430 08:00:14.156908 14073 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:35793 every 8 connection(s)
I20260430 08:00:14.157768 13945 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:00:14.164776 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 13945
I20260430 08:00:14.164897 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:14.165140 13690 external_mini_cluster.cc:1468] Setting key 7056299a0000a7de13d70a80ce60de9e
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536014, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:14.170776 13783 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:14.159402 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:41487 (local address 127.13.94.190:34499)
0430 08:00:14.159558 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:14.159561 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:14.160137 (+ 576us) server_negotiation.cc:408] Connection header received
0430 08:00:14.161143 (+ 1006us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:14.161147 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:14.161210 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:14.161308 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:14.162809 (+ 1501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.163318 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:14.163973 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.164149 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:14.166878 (+ 2729us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:14.166899 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:14.166901 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:14.166945 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:14.168589 (+ 1644us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:14.169257 (+ 668us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:14.169260 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:14.169262 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:14.169309 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:14.169657 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:14.169659 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:14.169660 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:14.169806 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:14.169883 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:14.170519 (+ 636us) server_negotiation.cc:300] Negotiation successful
0430 08:00:14.170637 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260430 08:00:14.171703 14076 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:14.159617 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:34499 (local address 127.13.94.130:41487)
0430 08:00:14.160001 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:14.160032 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:14.160937 (+ 905us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:14.161456 (+ 519us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:14.161465 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:14.161840 (+ 375us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:14.162668 (+ 828us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:14.162679 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.163450 (+ 771us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:14.163453 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:14.163855 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:14.163862 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.164080 (+ 218us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:14.164775 (+ 695us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:14.164807 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:14.166733 (+ 1926us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:14.168767 (+ 2034us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:14.168773 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:14.168788 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:14.169138 (+ 350us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:14.169411 (+ 273us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:14.169418 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:14.169423 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:14.169558 (+ 135us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:14.169911 (+ 353us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:14.169920 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:14.170228 (+ 308us) client_negotiation.cc:770] Sending connection context
0430 08:00:14.170479 (+ 251us) client_negotiation.cc:241] Negotiation successful
0430 08:00:14.170746 (+ 267us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260430 08:00:14.172869 14074 heartbeater.cc:344] Connected to a master server at 127.13.94.190:34499
I20260430 08:00:14.173128 14074 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:14.173630 14074 heartbeater.cc:507] Master 127.13.94.190:34499 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
I20260430 08:00:14.174779 13745 ts_manager.cc:194] Registered new tserver with Master: e0d525d0162a4ae8bd19af88ed6b1637 (127.13.94.130:35793)
I20260430 08:00:14.175388 13745 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:41487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:14.215824 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:34499
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:14.323141 14081 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:14.323410 14081 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:14.323525 14081 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:14.327044 14081 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:14.327152 14081 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:14.327265 14081 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:14.331823 14081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:34499
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:14.333025 14081 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:14.333916 14081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:14.340623 14087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:14.340787 14081 server_base.cc:1061] running on GCE node
W20260430 08:00:14.340636 14089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:14.340979 14086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:14.341413 14081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:14.342036 14081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:14.343230 14081 hybrid_clock.cc:648] HybridClock initialized: now 1777536014343199 us; error 44 us; skew 500 ppm
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536014, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:14.346199 14081 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:00:14.347211 14081 webserver.cc:492] Webserver started at http://127.13.94.131:46297/ using document root <none> and password file <none>
I20260430 08:00:14.347719 14081 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:14.347765 14081 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:14.347918 14081 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:14.349732 14081 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "cfe1d6deb9a44c25851c20a226bf5459"
format_stamp: "Formatted at 2026-04-30 08:00:14 on dist-test-slave-9bxp"
server_key: "3d0bbb1f3045fa06dd80ffe3d7402033"
server_key_iv: "0662ba395b85259f76a89befde579b5d"
server_key_version: "encryptionkey@0"
I20260430 08:00:14.350173 14081 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "cfe1d6deb9a44c25851c20a226bf5459"
format_stamp: "Formatted at 2026-04-30 08:00:14 on dist-test-slave-9bxp"
server_key: "3d0bbb1f3045fa06dd80ffe3d7402033"
server_key_iv: "0662ba395b85259f76a89befde579b5d"
server_key_version: "encryptionkey@0"
I20260430 08:00:14.353500 14081 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:00:14.355706 14096 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:14.356805 14081 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:14.356904 14081 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "cfe1d6deb9a44c25851c20a226bf5459"
format_stamp: "Formatted at 2026-04-30 08:00:14 on dist-test-slave-9bxp"
server_key: "3d0bbb1f3045fa06dd80ffe3d7402033"
server_key_iv: "0662ba395b85259f76a89befde579b5d"
server_key_version: "encryptionkey@0"
I20260430 08:00:14.357005 14081 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:14.368750 14081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:14.371222 14081 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:14.371428 14081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:14.371933 14081 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:14.372830 14081 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:14.372879 14081 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:14.372917 14081 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:14.372938 14081 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:14.381860 14081 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:45831
I20260430 08:00:14.381882 14209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:45831 every 8 connection(s)
I20260430 08:00:14.382807 14081 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536014, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:14.391878 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 14081
I20260430 08:00:14.391984 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:14.392203 13690 external_mini_cluster.cc:1468] Setting key 172191351a6fd02cf7aad5c9fd6a0a19
I20260430 08:00:14.395735 13783 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:14.384648 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:33073 (local address 127.13.94.190:34499)
0430 08:00:14.384829 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:14.384834 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:14.385497 (+ 663us) server_negotiation.cc:408] Connection header received
0430 08:00:14.386379 (+ 882us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:14.386382 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:14.386433 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:14.386511 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:14.388145 (+ 1634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.388634 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:14.389347 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.389476 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:14.391936 (+ 2460us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:14.391985 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:14.391999 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:14.392037 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:14.393804 (+ 1767us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:14.394338 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:14.394342 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:14.394344 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:14.394397 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:14.394773 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:14.394778 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:14.394780 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:14.394971 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:14.395059 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:14.395509 (+ 450us) server_negotiation.cc:300] Negotiation successful
0430 08:00:14.395612 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260430 08:00:14.396428 14212 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:14.384957 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:34499 (local address 127.13.94.131:33073)
0430 08:00:14.385358 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:14.385390 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:14.386177 (+ 787us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:14.386649 (+ 472us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:14.386656 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:14.387087 (+ 431us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:14.387930 (+ 843us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:14.387942 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.388841 (+ 899us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:14.388845 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:14.389247 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:14.389253 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.389410 (+ 157us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:14.389953 (+ 543us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:14.389972 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:14.391750 (+ 1778us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:14.393926 (+ 2176us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:14.393933 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:14.393949 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:14.394233 (+ 284us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:14.394490 (+ 257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:14.394493 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:14.394494 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:14.394648 (+ 154us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:14.395072 (+ 424us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:14.395077 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:14.395297 (+ 220us) client_negotiation.cc:770] Sending connection context
0430 08:00:14.395473 (+ 176us) client_negotiation.cc:241] Negotiation successful
0430 08:00:14.395666 (+ 193us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
I20260430 08:00:14.397504 14210 heartbeater.cc:344] Connected to a master server at 127.13.94.190:34499
I20260430 08:00:14.397761 14210 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:14.398283 14210 heartbeater.cc:507] Master 127.13.94.190:34499 requested a full tablet report, sending...
I20260430 08:00:14.399314 13745 ts_manager.cc:194] Registered new tserver with Master: cfe1d6deb9a44c25851c20a226bf5459 (127.13.94.131:45831)
I20260430 08:00:14.399904 13745 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:33073
I20260430 08:00:14.406992 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:00:14.433362 14217 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:14.424643 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.13.94.129:38611)
0430 08:00:14.425072 (+ 429us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:14.425078 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:14.425101 (+ 23us) server_negotiation.cc:408] Connection header received
0430 08:00:14.425189 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:14.425193 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:14.425359 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:14.425480 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:14.426344 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.427054 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:14.427692 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.427892 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:14.430115 (+ 2223us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:14.430161 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:14.430172 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:14.430201 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:14.431835 (+ 1634us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:14.432249 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:14.432253 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:14.432255 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:14.432300 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:14.432483 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:14.432485 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:14.432486 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:14.432654 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:14.432883 (+ 229us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:14.433094 (+ 211us) server_negotiation.cc:300] Negotiation successful
0430 08:00:14.433219 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":106,"threads_started":1}
W20260430 08:00:14.434267 13852 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52560
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[13697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[13697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:00:14.459082 14217 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:14.451197 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.13.94.129:38611)
0430 08:00:14.451363 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:14.451370 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:14.451430 (+ 60us) server_negotiation.cc:408] Connection header received
0430 08:00:14.451610 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:14.451613 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:14.451659 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:14.451807 (+ 148us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:14.452632 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.453212 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:14.453869 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:14.454093 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:14.456296 (+ 2203us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:14.456315 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:14.456318 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:14.456346 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:14.457800 (+ 1454us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:14.458159 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:14.458162 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:14.458164 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:14.458205 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:14.458426 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:14.458429 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:14.458430 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:14.458548 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:14.458642 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:14.458863 (+ 221us) server_negotiation.cc:300] Negotiation successful
0430 08:00:14.458953 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260430 08:00:14.460139 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 13809
I20260430 08:00:14.466307 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 13945
I20260430 08:00:14.474536 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 14081
I20260430 08:00:14.480603 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 13713
2026-04-30T08:00:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3176 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[14224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[14224](info): set up 2 sockets
Apr 30 08:00:14 dist-test-slave-9bxp krb5kdc[14224](info): commencing operation
krb5kdc: starting...
W20260430 08:00:16.519174 13690 mini_kdc.cc:121] Time spent starting KDC: real 2.015s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:00:16 dist-test-slave-9bxp krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:16.676283 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:33821
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:16.780054 14240 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:16.780364 14240 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:16.780445 14240 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:16.783905 14240 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:16.784024 14240 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:16.784070 14240 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:16.784132 14240 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:16.784176 14240 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:16.788666 14240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:33821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:33821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:16.789948 14240 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:16.790887 14240 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:16.796965 14246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:16.796974 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:16.796973 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:16.797142 14240 server_base.cc:1061] running on GCE node
I20260430 08:00:16.797760 14240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:16.798717 14240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:16.799937 14240 hybrid_clock.cc:648] HybridClock initialized: now 1777536016799916 us; error 42 us; skew 500 ppm
Apr 30 08:00:16 dist-test-slave-9bxp krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536016, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:16.802839 14240 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:16.804004 14240 webserver.cc:492] Webserver started at http://127.13.94.190:35441/ using document root <none> and password file <none>
I20260430 08:00:16.804553 14240 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:16.804627 14240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:16.804874 14240 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:16.806545 14240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "5ebf3b0f09b349fea440a202e9e382fd"
format_stamp: "Formatted at 2026-04-30 08:00:16 on dist-test-slave-9bxp"
server_key: "2bf66dd934c4d30a54ffb01f69295563"
server_key_iv: "711b300e2e73781c77117b90204e6f4d"
server_key_version: "encryptionkey@0"
I20260430 08:00:16.807039 14240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "5ebf3b0f09b349fea440a202e9e382fd"
format_stamp: "Formatted at 2026-04-30 08:00:16 on dist-test-slave-9bxp"
server_key: "2bf66dd934c4d30a54ffb01f69295563"
server_key_iv: "711b300e2e73781c77117b90204e6f4d"
server_key_version: "encryptionkey@0"
I20260430 08:00:16.810474 14240 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:16.812846 14255 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:16.813869 14240 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:00:16.814035 14240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "5ebf3b0f09b349fea440a202e9e382fd"
format_stamp: "Formatted at 2026-04-30 08:00:16 on dist-test-slave-9bxp"
server_key: "2bf66dd934c4d30a54ffb01f69295563"
server_key_iv: "711b300e2e73781c77117b90204e6f4d"
server_key_version: "encryptionkey@0"
I20260430 08:00:16.814146 14240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:16.829313 14240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:16.832000 14240 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:16.832207 14240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:16.839704 14240 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:33821
I20260430 08:00:16.839699 14307 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:33821 every 8 connection(s)
I20260430 08:00:16.840907 14240 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:16.842121 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 14240
I20260430 08:00:16.842245 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:16.842543 13690 external_mini_cluster.cc:1468] Setting key 01dc47f31eeef9207ed59a3543037f49
I20260430 08:00:16.844324 14308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Apr 30 08:00:16 dist-test-slave-9bxp krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:16.850142 14308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd: Bootstrap starting.
I20260430 08:00:16.852399 14308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:16.853173 14308 log.cc:826] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:16.855105 14308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd: No bootstrap required, opened a new log
I20260430 08:00:16.855176 14311 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:16.843923 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.13.94.190:33821)
0430 08:00:16.844336 (+ 413us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:16.844346 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:16.844402 (+ 56us) server_negotiation.cc:408] Connection header received
0430 08:00:16.845124 (+ 722us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:16.845144 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:16.845410 (+ 266us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:16.845719 (+ 309us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:16.846599 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:16.847361 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:16.848028 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:16.848316 (+ 288us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:16.850784 (+ 2468us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:16.850843 (+ 59us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:16.850855 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:16.850883 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:16.852586 (+ 1703us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:16.853032 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:16.853044 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:16.853049 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:16.853119 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:16.853378 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:16.853381 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:16.853384 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:16.853793 (+ 409us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:16.854005 (+ 212us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:16.854335 (+ 330us) server_negotiation.cc:300] Negotiation successful
0430 08:00:16.854587 (+ 252us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":142,"threads_started":1}
I20260430 08:00:16.857715 14308 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ebf3b0f09b349fea440a202e9e382fd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33821 } }
I20260430 08:00:16.858076 14308 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:16.858161 14308 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ebf3b0f09b349fea440a202e9e382fd, State: Initialized, Role: FOLLOWER
I20260430 08:00:16.858542 14308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ebf3b0f09b349fea440a202e9e382fd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33821 } }
I20260430 08:00:16.858688 14308 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:16.858767 14308 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:16.858872 14308 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:16.859754 14308 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ebf3b0f09b349fea440a202e9e382fd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33821 } }
I20260430 08:00:16.860110 14308 leader_election.cc:304] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ebf3b0f09b349fea440a202e9e382fd; no voters:
I20260430 08:00:16.860395 14308 leader_election.cc:290] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:16.860527 14313 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:16.860792 14313 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [term 1 LEADER]: Becoming Leader. State: Replica: 5ebf3b0f09b349fea440a202e9e382fd, State: Running, Role: LEADER
I20260430 08:00:16.861114 14313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ebf3b0f09b349fea440a202e9e382fd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33821 } }
I20260430 08:00:16.861601 14308 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:16.862859 14314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ebf3b0f09b349fea440a202e9e382fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ebf3b0f09b349fea440a202e9e382fd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33821 } } }
I20260430 08:00:16.862998 14314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:16.862169 14313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ebf3b0f09b349fea440a202e9e382fd. Latest consensus state: current_term: 1 leader_uuid: "5ebf3b0f09b349fea440a202e9e382fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ebf3b0f09b349fea440a202e9e382fd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33821 } } }
I20260430 08:00:16.863054 14313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:16.863368 14322 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:16.865981 14322 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:16.871006 14322 catalog_manager.cc:1357] Generated new cluster ID: 3b5ce6af41bb4763941b9f2f62228d65
I20260430 08:00:16.871085 14322 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:16.889505 14322 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:16.890739 14322 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:16.902976 14322 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5ebf3b0f09b349fea440a202e9e382fd: Generated new TSK 0
I20260430 08:00:16.903771 14322 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:16.960724 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:33821
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:17.066386 14336 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:17.066680 14336 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:17.066779 14336 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:17.070370 14336 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:17.070492 14336 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:17.070616 14336 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:17.075174 14336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:33821
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:17.076386 14336 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:17.077302 14336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:17.084314 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:17.084324 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:17.084342 14344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:17.084591 14336 server_base.cc:1061] running on GCE node
I20260430 08:00:17.085060 14336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:17.085599 14336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:17.086827 14336 hybrid_clock.cc:648] HybridClock initialized: now 1777536017086824 us; error 46 us; skew 500 ppm
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:17.089717 14336 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:00:17.090914 14336 webserver.cc:492] Webserver started at http://127.13.94.129:39591/ using document root <none> and password file <none>
I20260430 08:00:17.091507 14336 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:17.091584 14336 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:17.091818 14336 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:17.093694 14336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "5783aa27e41c46faa2ee7dfa60dd281c"
format_stamp: "Formatted at 2026-04-30 08:00:17 on dist-test-slave-9bxp"
server_key: "fbf349440ecf68689f88d8d130418188"
server_key_iv: "2e597775b1f49b744d959423c794d29d"
server_key_version: "encryptionkey@0"
I20260430 08:00:17.094214 14336 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "5783aa27e41c46faa2ee7dfa60dd281c"
format_stamp: "Formatted at 2026-04-30 08:00:17 on dist-test-slave-9bxp"
server_key: "fbf349440ecf68689f88d8d130418188"
server_key_iv: "2e597775b1f49b744d959423c794d29d"
server_key_version: "encryptionkey@0"
I20260430 08:00:17.097828 14336 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260430 08:00:17.100311 14351 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:17.101528 14336 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:17.101689 14336 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "5783aa27e41c46faa2ee7dfa60dd281c"
format_stamp: "Formatted at 2026-04-30 08:00:17 on dist-test-slave-9bxp"
server_key: "fbf349440ecf68689f88d8d130418188"
server_key_iv: "2e597775b1f49b744d959423c794d29d"
server_key_version: "encryptionkey@0"
I20260430 08:00:17.101816 14336 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:17.114337 14336 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:17.117292 14336 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:17.117506 14336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:17.118170 14336 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:17.119105 14336 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:17.119177 14336 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:17.119248 14336 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:17.119297 14336 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:17.129014 14336 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:32855
I20260430 08:00:17.129040 14464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:32855 every 8 connection(s)
I20260430 08:00:17.130033 14336 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:00:17.137341 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 14336
I20260430 08:00:17.137470 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:17.137773 13690 external_mini_cluster.cc:1468] Setting key d1d9636e24e54242b5a2f2fb1a6baba2
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:17.143956 14311 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.131906 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:60475 (local address 127.13.94.190:33821)
0430 08:00:17.132079 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.132084 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.133126 (+ 1042us) server_negotiation.cc:408] Connection header received
0430 08:00:17.133927 (+ 801us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.133930 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.133980 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.134068 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:17.135676 (+ 1608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.136247 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.136943 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.137101 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.139871 (+ 2770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:17.139893 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:17.139900 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:17.139942 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:17.141616 (+ 1674us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.142203 (+ 587us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.142207 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.142209 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.142266 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.142689 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.142693 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.142695 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.142916 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:17.143066 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.143645 (+ 579us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.143786 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 08:00:17.144622 14467 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.132245 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:33821 (local address 127.13.94.129:60475)
0430 08:00:17.132945 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:17.132979 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:17.133705 (+ 726us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:17.134225 (+ 520us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:17.134233 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:17.134616 (+ 383us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:17.135504 (+ 888us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.135517 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.136375 (+ 858us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.136378 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:17.136804 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.136811 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.136987 (+ 176us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.137788 (+ 801us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:17.137865 (+ 77us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:17.139705 (+ 1840us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:17.141744 (+ 2039us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:17.141752 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:17.141767 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:17.142043 (+ 276us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:17.142369 (+ 326us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:17.142373 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:17.142375 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:17.142496 (+ 121us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:17.143066 (+ 570us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:17.143074 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:17.143375 (+ 301us) client_negotiation.cc:770] Sending connection context
0430 08:00:17.143612 (+ 237us) client_negotiation.cc:241] Negotiation successful
0430 08:00:17.143869 (+ 257us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":112,"threads_started":1}
I20260430 08:00:17.146142 14465 heartbeater.cc:344] Connected to a master server at 127.13.94.190:33821
I20260430 08:00:17.146423 14465 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
I20260430 08:00:17.147004 14465 heartbeater.cc:507] Master 127.13.94.190:33821 requested a full tablet report, sending...
I20260430 08:00:17.148659 14272 ts_manager.cc:194] Registered new tserver with Master: 5783aa27e41c46faa2ee7dfa60dd281c (127.13.94.129:32855)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
I20260430 08:00:17.150120 14272 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:60475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:17.192263 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:33821
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:17.295125 14472 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:17.295377 14472 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:17.295439 14472 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:17.298987 14472 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:17.299072 14472 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:17.299170 14472 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:17.303774 14472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:33821
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:17.304991 14472 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:17.305858 14472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:17.313014 14477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:17.312979 14480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:17.312983 14478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:17.313302 14472 server_base.cc:1061] running on GCE node
I20260430 08:00:17.313653 14472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:17.314251 14472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:17.315421 14472 hybrid_clock.cc:648] HybridClock initialized: now 1777536017315391 us; error 41 us; skew 500 ppm
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:17.318459 14472 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:00:17.319957 14472 webserver.cc:492] Webserver started at http://127.13.94.130:41585/ using document root <none> and password file <none>
I20260430 08:00:17.320566 14472 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:17.320618 14472 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:17.320909 14472 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:17.322722 14472 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "85b31cc503314465acac67599c11124c"
format_stamp: "Formatted at 2026-04-30 08:00:17 on dist-test-slave-9bxp"
server_key: "f8cac0d852b4378cef722ceda042dff5"
server_key_iv: "4902812bc36bc17e1a8b92004f452a91"
server_key_version: "encryptionkey@0"
I20260430 08:00:17.323239 14472 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "85b31cc503314465acac67599c11124c"
format_stamp: "Formatted at 2026-04-30 08:00:17 on dist-test-slave-9bxp"
server_key: "f8cac0d852b4378cef722ceda042dff5"
server_key_iv: "4902812bc36bc17e1a8b92004f452a91"
server_key_version: "encryptionkey@0"
I20260430 08:00:17.326794 14472 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260430 08:00:17.329044 14487 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:17.330199 14472 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:00:17.330341 14472 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "85b31cc503314465acac67599c11124c"
format_stamp: "Formatted at 2026-04-30 08:00:17 on dist-test-slave-9bxp"
server_key: "f8cac0d852b4378cef722ceda042dff5"
server_key_iv: "4902812bc36bc17e1a8b92004f452a91"
server_key_version: "encryptionkey@0"
I20260430 08:00:17.330451 14472 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:17.344244 14472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:17.347208 14472 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:17.347417 14472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:17.348004 14472 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:17.348978 14472 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:17.349035 14472 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:17.349100 14472 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:17.349131 14472 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:17.358494 14472 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:43849
I20260430 08:00:17.358537 14600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:43849 every 8 connection(s)
I20260430 08:00:17.359503 14472 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:17.368427 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 14472
I20260430 08:00:17.368538 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:17.368835 13690 external_mini_cluster.cc:1468] Setting key d2e0eaf2789e1da6c55806c78a68f5df
I20260430 08:00:17.372180 14311 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.361286 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:33817 (local address 127.13.94.190:33821)
0430 08:00:17.361435 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.361439 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.362173 (+ 734us) server_negotiation.cc:408] Connection header received
0430 08:00:17.363186 (+ 1013us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.363191 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.363257 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.363376 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:17.365064 (+ 1688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.365582 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.366228 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.366411 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.368585 (+ 2174us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:17.368633 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:17.368636 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:17.368667 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:17.370315 (+ 1648us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.370789 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.370793 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.370795 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.370851 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.371151 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.371155 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.371157 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.371306 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:17.371410 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.371896 (+ 486us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.372024 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260430 08:00:17.372844 14603 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.361572 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:33821 (local address 127.13.94.130:33817)
0430 08:00:17.362029 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:17.362069 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:17.362938 (+ 869us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:17.363550 (+ 612us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:17.363558 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:17.363997 (+ 439us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:17.364910 (+ 913us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.364924 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.365708 (+ 784us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.365714 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:17.366109 (+ 395us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.366115 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.366331 (+ 216us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.366913 (+ 582us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:17.366930 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:17.368429 (+ 1499us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:17.370430 (+ 2001us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:17.370437 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:17.370450 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:17.370686 (+ 236us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:17.370943 (+ 257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:17.370947 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:17.370949 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:17.371050 (+ 101us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:17.371411 (+ 361us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:17.371418 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:17.371643 (+ 225us) client_negotiation.cc:770] Sending connection context
0430 08:00:17.371843 (+ 200us) client_negotiation.cc:241] Negotiation successful
0430 08:00:17.372074 (+ 231us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":97,"threads_started":1}
I20260430 08:00:17.373987 14601 heartbeater.cc:344] Connected to a master server at 127.13.94.190:33821
I20260430 08:00:17.374238 14601 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:17.374737 14601 heartbeater.cc:507] Master 127.13.94.190:33821 requested a full tablet report, sending...
I20260430 08:00:17.375885 14272 ts_manager.cc:194] Registered new tserver with Master: 85b31cc503314465acac67599c11124c (127.13.94.130:43849)
I20260430 08:00:17.376688 14272 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:33817
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:17.420969 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:33821
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:17.530227 14608 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:17.530505 14608 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:17.530606 14608 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:17.534440 14608 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:17.534559 14608 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:17.534672 14608 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:17.539214 14608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:33821
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:17.540395 14608 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:17.541364 14608 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:17.548206 14614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:17.548385 14608 server_base.cc:1061] running on GCE node
W20260430 08:00:17.548254 14613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:17.548251 14616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:17.548913 14608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:17.549535 14608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:17.550724 14608 hybrid_clock.cc:648] HybridClock initialized: now 1777536017550700 us; error 46 us; skew 500 ppm
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:17.553704 14608 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:00:17.554873 14608 webserver.cc:492] Webserver started at http://127.13.94.131:46265/ using document root <none> and password file <none>
I20260430 08:00:17.555544 14608 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:17.555616 14608 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:17.555825 14608 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:17.557791 14608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "f64f930ce45743b7a627368ae3d6f817"
format_stamp: "Formatted at 2026-04-30 08:00:17 on dist-test-slave-9bxp"
server_key: "802c6573a0d4e31c6a959859a420ea11"
server_key_iv: "ddf34d0a3ba4e9e5f9f7c470e4cd4bc7"
server_key_version: "encryptionkey@0"
I20260430 08:00:17.558310 14608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "f64f930ce45743b7a627368ae3d6f817"
format_stamp: "Formatted at 2026-04-30 08:00:17 on dist-test-slave-9bxp"
server_key: "802c6573a0d4e31c6a959859a420ea11"
server_key_iv: "ddf34d0a3ba4e9e5f9f7c470e4cd4bc7"
server_key_version: "encryptionkey@0"
I20260430 08:00:17.561735 14608 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260430 08:00:17.564069 14623 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:17.565208 14608 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:17.565344 14608 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "f64f930ce45743b7a627368ae3d6f817"
format_stamp: "Formatted at 2026-04-30 08:00:17 on dist-test-slave-9bxp"
server_key: "802c6573a0d4e31c6a959859a420ea11"
server_key_iv: "ddf34d0a3ba4e9e5f9f7c470e4cd4bc7"
server_key_version: "encryptionkey@0"
I20260430 08:00:17.565454 14608 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:17.594573 14608 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:17.597918 14608 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:17.598143 14608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:17.598726 14608 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:17.599666 14608 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:17.599737 14608 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:17.599807 14608 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:17.599859 14608 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:17.609555 14608 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:46227
I20260430 08:00:17.609589 14736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:46227 every 8 connection(s)
I20260430 08:00:17.610603 14608 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:00:17.617092 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 14608
I20260430 08:00:17.617255 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:17.617512 13690 external_mini_cluster.cc:1468] Setting key aa064f598afec93640bfb2738e0ac03b
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:17.623582 14311 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.612383 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:35395 (local address 127.13.94.190:33821)
0430 08:00:17.612522 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.612526 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.613270 (+ 744us) server_negotiation.cc:408] Connection header received
0430 08:00:17.614185 (+ 915us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.614189 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.614249 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.614328 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:17.615878 (+ 1550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.616413 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.617151 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.617333 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.620030 (+ 2697us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:17.620048 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:17.620052 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:17.620091 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:17.621594 (+ 1503us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.622171 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.622176 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.622178 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.622231 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.622533 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.622536 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.622538 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.622693 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:17.622800 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.623290 (+ 490us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.623409 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260430 08:00:17.624210 14739 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.612683 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:33821 (local address 127.13.94.131:35395)
0430 08:00:17.613131 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:17.613165 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:17.613962 (+ 797us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:17.614483 (+ 521us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:17.614491 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:17.614866 (+ 375us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:17.615717 (+ 851us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.615729 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.616543 (+ 814us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.616554 (+ 11us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:17.617022 (+ 468us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.617029 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.617260 (+ 231us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.617915 (+ 655us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:17.617940 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:17.619839 (+ 1899us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:17.621773 (+ 1934us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:17.621780 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:17.621793 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:17.622052 (+ 259us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:17.622325 (+ 273us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:17.622328 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:17.622331 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:17.622428 (+ 97us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:17.622798 (+ 370us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:17.622804 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:17.623035 (+ 231us) client_negotiation.cc:770] Sending connection context
0430 08:00:17.623258 (+ 223us) client_negotiation.cc:241] Negotiation successful
0430 08:00:17.623489 (+ 231us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260430 08:00:17.625392 14737 heartbeater.cc:344] Connected to a master server at 127.13.94.190:33821
I20260430 08:00:17.625685 14737 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:17.626247 14737 heartbeater.cc:507] Master 127.13.94.190:33821 requested a full tablet report, sending...
I20260430 08:00:17.627302 14272 ts_manager.cc:194] Registered new tserver with Master: f64f930ce45743b7a627368ae3d6f817 (127.13.94.131:46227)
I20260430 08:00:17.627961 14272 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:35395
I20260430 08:00:17.632045 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 08:00:17.641486 14311 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.634997 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.13.94.190:33821)
0430 08:00:17.635143 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.635147 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.635271 (+ 124us) server_negotiation.cc:408] Connection header received
0430 08:00:17.635371 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.635373 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.635415 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.635494 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:17.636296 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.636857 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.637499 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.637643 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.638668 (+ 1025us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:17.638682 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:17.638684 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:17.638707 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:17.640027 (+ 1320us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.640426 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.640429 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.640431 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.640487 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.640682 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.640684 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.640686 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.640925 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:17.641033 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.641277 (+ 244us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.641366 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260430 08:00:17.646005 14272 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:00:17.648350 14272 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:17.662111 14750 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.657579 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:46227 (local address 127.0.0.1:33854)
0430 08:00:17.657992 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:17.658009 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:17.658123 (+ 114us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:17.658993 (+ 870us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:17.658997 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:17.659011 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:17.659354 (+ 343us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.659367 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.660844 (+ 1477us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.660849 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:17.661678 (+ 829us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.661687 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.661798 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.661819 (+ 21us) client_negotiation.cc:770] Sending connection context
0430 08:00:17.661872 (+ 53us) client_negotiation.cc:241] Negotiation successful
0430 08:00:17.661914 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":105,"threads_started":1}
I20260430 08:00:17.662676 14751 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.658002 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:43849 (local address 127.0.0.1:53456)
0430 08:00:17.658911 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:17.658927 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:17.659005 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:17.659467 (+ 462us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:17.659474 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:17.659509 (+ 35us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:17.659775 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.659784 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.661375 (+ 1591us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.661381 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:17.662320 (+ 939us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.662329 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.662442 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.662456 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 08:00:17.662504 (+ 48us) client_negotiation.cc:241] Negotiation successful
0430 08:00:17.662553 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":99,"threads_started":1}
I20260430 08:00:17.662931 14752 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.657989 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.13.94.131:46227)
0430 08:00:17.658547 (+ 558us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.658553 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.658574 (+ 21us) server_negotiation.cc:408] Connection header received
0430 08:00:17.658657 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.658662 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.658835 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.659478 (+ 643us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:17.659633 (+ 155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.660739 (+ 1106us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.661955 (+ 1216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.662513 (+ 558us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.662627 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.662690 (+ 63us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.662786 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":166,"threads_started":1}
I20260430 08:00:17.663223 14749 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.657546 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.13.94.129:32855)
0430 08:00:17.658403 (+ 857us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.658411 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.658435 (+ 24us) server_negotiation.cc:408] Connection header received
0430 08:00:17.658531 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.658538 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.658755 (+ 217us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.659223 (+ 468us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:17.659557 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.660420 (+ 863us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.662185 (+ 1765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.662763 (+ 578us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.662870 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.662961 (+ 91us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.663053 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":99,"threads_started":1}
I20260430 08:00:17.662112 14748 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.657283 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:32855 (local address 127.0.0.1:58596)
0430 08:00:17.657930 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:17.657970 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:17.658116 (+ 146us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:17.658939 (+ 823us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:17.658946 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:17.658971 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:17.659380 (+ 409us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.659389 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.660553 (+ 1164us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.660558 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:17.661517 (+ 959us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.661527 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.661643 (+ 116us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.661702 (+ 59us) client_negotiation.cc:770] Sending connection context
0430 08:00:17.661809 (+ 107us) client_negotiation.cc:241] Negotiation successful
0430 08:00:17.661898 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":90,"threads_started":1}
I20260430 08:00:17.664036 14753 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.658402 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53456 (local address 127.13.94.130:43849)
0430 08:00:17.658746 (+ 344us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.658752 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.659091 (+ 339us) server_negotiation.cc:408] Connection header received
0430 08:00:17.659176 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.659181 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.659317 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.659454 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:17.659902 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.661233 (+ 1331us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.662888 (+ 1655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.663487 (+ 599us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.663584 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.663644 (+ 60us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.663894 (+ 250us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260430 08:00:17.666285 14535 tablet_service.cc:1511] Processing CreateTablet for tablet 36fbaff18ed24c69bd57c0610da557ce (DEFAULT_TABLE table=test-table [id=3dc0a82c5d5847378ccf701fafd16c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:17.666373 14399 tablet_service.cc:1511] Processing CreateTablet for tablet 36fbaff18ed24c69bd57c0610da557ce (DEFAULT_TABLE table=test-table [id=3dc0a82c5d5847378ccf701fafd16c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:17.666292 14671 tablet_service.cc:1511] Processing CreateTablet for tablet 36fbaff18ed24c69bd57c0610da557ce (DEFAULT_TABLE table=test-table [id=3dc0a82c5d5847378ccf701fafd16c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:17.667187 14535 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fbaff18ed24c69bd57c0610da557ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:17.667212 14399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fbaff18ed24c69bd57c0610da557ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:17.667212 14671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fbaff18ed24c69bd57c0610da557ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:17.672406 14754 tablet_bootstrap.cc:492] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c: Bootstrap starting.
I20260430 08:00:17.674008 14755 tablet_bootstrap.cc:492] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817: Bootstrap starting.
I20260430 08:00:17.674316 14756 tablet_bootstrap.cc:492] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c: Bootstrap starting.
I20260430 08:00:17.674773 14754 tablet_bootstrap.cc:654] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:17.675638 14754 log.cc:826] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:17.676185 14755 tablet_bootstrap.cc:654] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:17.676322 14756 tablet_bootstrap.cc:654] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:17.677070 14756 log.cc:826] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:17.677070 14755 log.cc:826] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:17.677800 14754 tablet_bootstrap.cc:492] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c: No bootstrap required, opened a new log
I20260430 08:00:17.678081 14754 ts_tablet_manager.cc:1403] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260430 08:00:17.678941 14756 tablet_bootstrap.cc:492] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c: No bootstrap required, opened a new log
I20260430 08:00:17.679078 14755 tablet_bootstrap.cc:492] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817: No bootstrap required, opened a new log
I20260430 08:00:17.679186 14756 ts_tablet_manager.cc:1403] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260430 08:00:17.679262 14755 ts_tablet_manager.cc:1403] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 08:00:17.681663 14756 raft_consensus.cc:359] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } }
I20260430 08:00:17.681907 14756 raft_consensus.cc:385] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:17.681990 14756 raft_consensus.cc:740] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5783aa27e41c46faa2ee7dfa60dd281c, State: Initialized, Role: FOLLOWER
I20260430 08:00:17.682101 14754 raft_consensus.cc:359] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } }
I20260430 08:00:17.682348 14754 raft_consensus.cc:385] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:17.682420 14754 raft_consensus.cc:740] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b31cc503314465acac67599c11124c, State: Initialized, Role: FOLLOWER
I20260430 08:00:17.682528 14756 consensus_queue.cc:260] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } }
I20260430 08:00:17.682829 14754 consensus_queue.cc:260] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } }
I20260430 08:00:17.682979 14755 raft_consensus.cc:359] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } }
I20260430 08:00:17.683164 14755 raft_consensus.cc:385] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:17.683231 14755 raft_consensus.cc:740] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f64f930ce45743b7a627368ae3d6f817, State: Initialized, Role: FOLLOWER
I20260430 08:00:17.683320 14465 heartbeater.cc:499] Master 127.13.94.190:33821 was elected leader, sending a full tablet report...
I20260430 08:00:17.683586 14756 ts_tablet_manager.cc:1434] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260430 08:00:17.683674 14755 consensus_queue.cc:260] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } }
I20260430 08:00:17.683933 14754 ts_tablet_manager.cc:1434] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260430 08:00:17.684312 14601 heartbeater.cc:499] Master 127.13.94.190:33821 was elected leader, sending a full tablet report...
I20260430 08:00:17.684576 14737 heartbeater.cc:499] Master 127.13.94.190:33821 was elected leader, sending a full tablet report...
I20260430 08:00:17.685204 14755 ts_tablet_manager.cc:1434] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260430 08:00:17.860664 14602 tablet.cc:2404] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:00:17.861444 14738 tablet.cc:2404] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:00:17.881302 14466 tablet.cc:2404] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:00:17.931361 14762 raft_consensus.cc:493] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:17.931553 14762 raft_consensus.cc:515] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } }
I20260430 08:00:17.932773 14762 leader_election.cc:290] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5783aa27e41c46faa2ee7dfa60dd281c (127.13.94.129:32855), 85b31cc503314465acac67599c11124c (127.13.94.130:43849)
I20260430 08:00:17.935962 14739 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.933038 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:43849 (local address 127.13.94.131:60611)
0430 08:00:17.933237 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:17.933255 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:17.933361 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:17.933694 (+ 333us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:17.933701 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:17.933722 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:17.934057 (+ 335us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.934065 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.934917 (+ 852us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.934922 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:17.935521 (+ 599us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.935527 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.935692 (+ 165us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.935712 (+ 20us) client_negotiation.cc:770] Sending connection context
0430 08:00:17.935769 (+ 57us) client_negotiation.cc:241] Negotiation successful
0430 08:00:17.935821 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260430 08:00:17.936049 14763 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.933074 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:32855 (local address 127.13.94.131:36961)
0430 08:00:17.933526 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:17.933537 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:17.933619 (+ 82us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:17.933928 (+ 309us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:17.933931 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:17.933950 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:17.934198 (+ 248us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.934203 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.934939 (+ 736us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.934945 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:17.935637 (+ 692us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:17.935644 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.935780 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.935795 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 08:00:17.935846 (+ 51us) client_negotiation.cc:241] Negotiation successful
0430 08:00:17.935894 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":121,"threads_started":1}
I20260430 08:00:17.936443 14753 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.933239 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:60611 (local address 127.13.94.130:43849)
0430 08:00:17.933376 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.933381 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.933426 (+ 45us) server_negotiation.cc:408] Connection header received
0430 08:00:17.933488 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.933492 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.933550 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.933662 (+ 112us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:17.934192 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.934803 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.935649 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.936153 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.936197 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.936250 (+ 53us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.936321 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260430 08:00:17.936512 14749 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.933211 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:36961 (local address 127.13.94.129:32855)
0430 08:00:17.933453 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.933457 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.933620 (+ 163us) server_negotiation.cc:408] Connection header received
0430 08:00:17.933756 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.933759 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.933823 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.933915 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:17.934321 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.934826 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.935766 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.936254 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.936295 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.936344 (+ 49us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.936391 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260430 08:00:17.937147 14555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36fbaff18ed24c69bd57c0610da557ce" candidate_uuid: "f64f930ce45743b7a627368ae3d6f817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b31cc503314465acac67599c11124c" is_pre_election: true
I20260430 08:00:17.937469 14555 raft_consensus.cc:2468] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f64f930ce45743b7a627368ae3d6f817 in term 0.
I20260430 08:00:17.937520 14419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36fbaff18ed24c69bd57c0610da557ce" candidate_uuid: "f64f930ce45743b7a627368ae3d6f817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" is_pre_election: true
I20260430 08:00:17.937764 14419 raft_consensus.cc:2468] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f64f930ce45743b7a627368ae3d6f817 in term 0.
I20260430 08:00:17.938059 14626 leader_election.cc:304] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b31cc503314465acac67599c11124c, f64f930ce45743b7a627368ae3d6f817; no voters:
I20260430 08:00:17.938354 14762 raft_consensus.cc:2804] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:17.938429 14762 raft_consensus.cc:493] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:17.938472 14762 raft_consensus.cc:3060] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:17.939460 14762 raft_consensus.cc:515] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } }
I20260430 08:00:17.939836 14762 leader_election.cc:290] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [CANDIDATE]: Term 1 election: Requested vote from peers 5783aa27e41c46faa2ee7dfa60dd281c (127.13.94.129:32855), 85b31cc503314465acac67599c11124c (127.13.94.130:43849)
I20260430 08:00:17.940251 14419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36fbaff18ed24c69bd57c0610da557ce" candidate_uuid: "f64f930ce45743b7a627368ae3d6f817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5783aa27e41c46faa2ee7dfa60dd281c"
I20260430 08:00:17.940258 14555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36fbaff18ed24c69bd57c0610da557ce" candidate_uuid: "f64f930ce45743b7a627368ae3d6f817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b31cc503314465acac67599c11124c"
I20260430 08:00:17.940390 14419 raft_consensus.cc:3060] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:17.940390 14555 raft_consensus.cc:3060] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:17.941730 14419 raft_consensus.cc:2468] T 36fbaff18ed24c69bd57c0610da557ce P 5783aa27e41c46faa2ee7dfa60dd281c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f64f930ce45743b7a627368ae3d6f817 in term 1.
I20260430 08:00:17.941737 14555 raft_consensus.cc:2468] T 36fbaff18ed24c69bd57c0610da557ce P 85b31cc503314465acac67599c11124c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f64f930ce45743b7a627368ae3d6f817 in term 1.
I20260430 08:00:17.942167 14624 leader_election.cc:304] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5783aa27e41c46faa2ee7dfa60dd281c, f64f930ce45743b7a627368ae3d6f817; no voters:
I20260430 08:00:17.942442 14762 raft_consensus.cc:2804] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:17.942683 14762 raft_consensus.cc:697] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [term 1 LEADER]: Becoming Leader. State: Replica: f64f930ce45743b7a627368ae3d6f817, State: Running, Role: LEADER
I20260430 08:00:17.943049 14762 consensus_queue.cc:237] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } }
I20260430 08:00:17.946528 14272 catalog_manager.cc:5671] T 36fbaff18ed24c69bd57c0610da557ce P f64f930ce45743b7a627368ae3d6f817 reported cstate change: term changed from 0 to 1, leader changed from <none> to f64f930ce45743b7a627368ae3d6f817 (127.13.94.131). New cstate: current_term: 1 leader_uuid: "f64f930ce45743b7a627368ae3d6f817" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5783aa27e41c46faa2ee7dfa60dd281c" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 32855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f64f930ce45743b7a627368ae3d6f817" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 46227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85b31cc503314465acac67599c11124c" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 43849 } health_report { overall_health: UNKNOWN } } }
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:00:17.965770 14749 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.956492 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.13.94.129:32855)
0430 08:00:17.956662 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.956669 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.956902 (+ 233us) server_negotiation.cc:408] Connection header received
0430 08:00:17.957019 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.957022 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.957073 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.957156 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:17.958153 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.958685 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.959376 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.959512 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.962125 (+ 2613us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:17.962175 (+ 50us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:17.962188 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:17.962218 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:17.964132 (+ 1914us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.964578 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.964583 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.964585 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.964634 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.964958 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.964961 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.964962 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.965139 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:17.965294 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.965521 (+ 227us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.965634 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:00:17.992106 14749 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:17.983532 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.13.94.129:32855)
0430 08:00:17.983737 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:17.983743 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:17.983940 (+ 197us) server_negotiation.cc:408] Connection header received
0430 08:00:17.984127 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:17.984131 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:17.984183 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:17.984281 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:17.985217 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.985698 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:17.986413 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:17.986624 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:17.989077 (+ 2453us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:17.989102 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:17.989104 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:17.989135 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:17.990494 (+ 1359us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.990947 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.990951 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.990953 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.991008 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:17.991261 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:17.991264 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:17.991267 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:17.991410 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:17.991563 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:17.991795 (+ 232us) server_negotiation.cc:300] Negotiation successful
0430 08:00:17.991914 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Apr 30 08:00:17 dist-test-slave-9bxp krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 08:00:18 dist-test-slave-9bxp krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:00:18.018244 14749 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:18.009630 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.13.94.129:32855)
0430 08:00:18.009792 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:18.009797 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:18.009889 (+ 92us) server_negotiation.cc:408] Connection header received
0430 08:00:18.010069 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:18.010075 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:18.010131 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:18.010226 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:18.011045 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:18.011519 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:18.012194 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:18.012406 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:18.014921 (+ 2515us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:18.014944 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:18.014946 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:18.014979 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:18.016558 (+ 1579us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:18.017083 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:18.017112 (+ 29us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:18.017114 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:18.017163 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:18.017413 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:18.017417 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:18.017419 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:18.017560 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:18.017706 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:18.017934 (+ 228us) server_negotiation.cc:300] Negotiation successful
0430 08:00:18.018055 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260430 08:00:18.019716 14379 tablet_service.cc:3018] Rejecting scan request for tablet 36fbaff18ed24c69bd57c0610da557ce: Uninitialized: clean time has not yet been initialized
I20260430 08:00:18.021592 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 14336
I20260430 08:00:18.027904 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 14472
I20260430 08:00:18.033988 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 14608
I20260430 08:00:18.040024 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 14240
2026-04-30T08:00:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3559 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:18 dist-test-slave-9bxp krb5kdc[14778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:18 dist-test-slave-9bxp krb5kdc[14778](info): set up 2 sockets
Apr 30 08:00:18 dist-test-slave-9bxp krb5kdc[14778](info): commencing operation
krb5kdc: starting...
W20260430 08:00:20.049220 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.985s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:00:20 dist-test-slave-9bxp krb5kdc[14778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:20.196807 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:44423
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:44423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:20.304608 14794 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:20.304965 14794 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:20.305075 14794 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:20.309255 14794 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:20.309360 14794 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260430 08:00:20.309394 14794 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:20.309448 14794 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:20.309489 14794 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:20.309523 14794 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:20.314419 14794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:44423
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:44423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:20.315830 14794 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:20.316931 14794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:20.323875 14800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:20.323913 14799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:20.323941 14802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:20.324501 14794 server_base.cc:1061] running on GCE node
I20260430 08:00:20.325126 14794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:20.326197 14794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:20.327380 14794 hybrid_clock.cc:648] HybridClock initialized: now 1777536020327343 us; error 56 us; skew 500 ppm
Apr 30 08:00:20 dist-test-slave-9bxp krb5kdc[14778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536020, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:20.330742 14794 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:20.331969 14794 webserver.cc:492] Webserver started at http://127.13.94.190:39519/ using document root <none> and password file <none>
I20260430 08:00:20.332548 14794 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:20.332599 14794 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:20.332866 14794 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:20.334661 14794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "c45804c8211f45679698e690f3c71d60"
format_stamp: "Formatted at 2026-04-30 08:00:20 on dist-test-slave-9bxp"
server_key: "2a361e8043d8d7976ec32dcea0da67c8"
server_key_iv: "8220ae5338dd1866e79fa42ae95144a1"
server_key_version: "encryptionkey@0"
I20260430 08:00:20.335156 14794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "c45804c8211f45679698e690f3c71d60"
format_stamp: "Formatted at 2026-04-30 08:00:20 on dist-test-slave-9bxp"
server_key: "2a361e8043d8d7976ec32dcea0da67c8"
server_key_iv: "8220ae5338dd1866e79fa42ae95144a1"
server_key_version: "encryptionkey@0"
I20260430 08:00:20.338979 14794 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 08:00:20.341477 14809 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:20.342705 14794 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:20.342903 14794 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "c45804c8211f45679698e690f3c71d60"
format_stamp: "Formatted at 2026-04-30 08:00:20 on dist-test-slave-9bxp"
server_key: "2a361e8043d8d7976ec32dcea0da67c8"
server_key_iv: "8220ae5338dd1866e79fa42ae95144a1"
server_key_version: "encryptionkey@0"
I20260430 08:00:20.343031 14794 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:20.359753 14794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:20.363315 14794 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:20.363533 14794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:20.372035 14794 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:44423
I20260430 08:00:20.372318 14871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:44423 every 8 connection(s)
I20260430 08:00:20.373502 14794 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:20.376574 14872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:20.381862 14872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60: Bootstrap starting.
I20260430 08:00:20.383464 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 14794
I20260430 08:00:20.383556 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:20.383805 13690 external_mini_cluster.cc:1468] Setting key 001c34aa69f2fdbd44e907e48af04de2
I20260430 08:00:20.384447 14872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:20.385288 14872 log.cc:826] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:20.387408 14872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60: No bootstrap required, opened a new log
I20260430 08:00:20.390520 14872 raft_consensus.cc:359] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c45804c8211f45679698e690f3c71d60" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 44423 } }
I20260430 08:00:20.390710 14872 raft_consensus.cc:385] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:20.390774 14872 raft_consensus.cc:740] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45804c8211f45679698e690f3c71d60, State: Initialized, Role: FOLLOWER
I20260430 08:00:20.391249 14872 consensus_queue.cc:260] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c45804c8211f45679698e690f3c71d60" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 44423 } }
I20260430 08:00:20.391418 14872 raft_consensus.cc:399] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:20.391505 14872 raft_consensus.cc:493] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:20.391620 14872 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [term 0 FOLLOWER]: Advancing to term 1
Apr 30 08:00:20 dist-test-slave-9bxp krb5kdc[14778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:20.393411 14872 raft_consensus.cc:515] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c45804c8211f45679698e690f3c71d60" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 44423 } }
I20260430 08:00:20.393798 14872 leader_election.cc:304] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c45804c8211f45679698e690f3c71d60; no voters:
I20260430 08:00:20.394147 14872 leader_election.cc:290] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:20.394235 14877 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:20.394418 14877 raft_consensus.cc:697] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [term 1 LEADER]: Becoming Leader. State: Replica: c45804c8211f45679698e690f3c71d60, State: Running, Role: LEADER
I20260430 08:00:20.394793 14877 consensus_queue.cc:237] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c45804c8211f45679698e690f3c71d60" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 44423 } }
I20260430 08:00:20.395484 14872 sys_catalog.cc:565] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:20.396669 14878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c45804c8211f45679698e690f3c71d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c45804c8211f45679698e690f3c71d60" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 44423 } } }
I20260430 08:00:20.396842 14878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:20.397131 14879 sys_catalog.cc:455] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c45804c8211f45679698e690f3c71d60. Latest consensus state: current_term: 1 leader_uuid: "c45804c8211f45679698e690f3c71d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c45804c8211f45679698e690f3c71d60" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 44423 } } }
I20260430 08:00:20.397222 14879 sys_catalog.cc:458] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:20.397284 14886 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:20.398366 14875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:20.385002 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.13.94.190:44423)
0430 08:00:20.385520 (+ 518us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:20.385532 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:20.385573 (+ 41us) server_negotiation.cc:408] Connection header received
0430 08:00:20.386268 (+ 695us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:20.386294 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:20.386643 (+ 349us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:20.387015 (+ 372us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:20.387852 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.389053 (+ 1201us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:20.389785 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.390087 (+ 302us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:20.392451 (+ 2364us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:20.392478 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:20.392495 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:20.392530 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:20.394701 (+ 2171us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.395248 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.395254 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.395259 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.395336 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.395601 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.395605 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.395607 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.396041 (+ 434us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:20.396227 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:20.397265 (+ 1038us) server_negotiation.cc:300] Negotiation successful
0430 08:00:20.397531 (+ 266us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":154,"threads_started":1}
I20260430 08:00:20.400682 14886 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:20.406474 14886 catalog_manager.cc:1357] Generated new cluster ID: 2b5cf5a9c2e6484893865d1b2b64b130
I20260430 08:00:20.406571 14886 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:20.449642 14886 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:20.450977 14886 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:20.461170 14886 catalog_manager.cc:6044] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60: Generated new TSK 0
I20260430 08:00:20.461871 14886 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:20.528736 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:44423
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:20.641501 14900 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:20.641798 14900 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:20.641906 14900 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:20.645383 14900 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:20.645519 14900 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:20.645614 14900 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 08:00:20.645720 14900 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:20.650527 14900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.94.190:44423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:20.651753 14900 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:20.652763 14900 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:20.659663 14906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:20.659701 14905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:20.659732 14900 server_base.cc:1061] running on GCE node
W20260430 08:00:20.659734 14908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:20.660501 14900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:20.661218 14900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:20.662408 14900 hybrid_clock.cc:648] HybridClock initialized: now 1777536020662376 us; error 43 us; skew 500 ppm
Apr 30 08:00:20 dist-test-slave-9bxp krb5kdc[14778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536020, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:20.665557 14900 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:00:20.666733 14900 webserver.cc:492] Webserver started at http://127.13.94.129:45521/ using document root <none> and password file <none>
I20260430 08:00:20.667332 14900 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:20.667412 14900 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:20.667623 14900 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:20.669497 14900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd"
format_stamp: "Formatted at 2026-04-30 08:00:20 on dist-test-slave-9bxp"
server_key: "75bcf63d2df730851d82971402ad3e28"
server_key_iv: "c56ea75eb2d856d2b061aee646d25e2b"
server_key_version: "encryptionkey@0"
I20260430 08:00:20.670008 14900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd"
format_stamp: "Formatted at 2026-04-30 08:00:20 on dist-test-slave-9bxp"
server_key: "75bcf63d2df730851d82971402ad3e28"
server_key_iv: "c56ea75eb2d856d2b061aee646d25e2b"
server_key_version: "encryptionkey@0"
I20260430 08:00:20.673756 14900 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:00:20.676287 14915 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:20.677531 14900 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:20.677690 14900 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd"
format_stamp: "Formatted at 2026-04-30 08:00:20 on dist-test-slave-9bxp"
server_key: "75bcf63d2df730851d82971402ad3e28"
server_key_iv: "c56ea75eb2d856d2b061aee646d25e2b"
server_key_version: "encryptionkey@0"
I20260430 08:00:20.677816 14900 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:20.689692 14900 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:20.693137 14900 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:20.693372 14900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:20.695060 14900 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:20.695142 14900 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:20.695209 14900 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:20.695253 14900 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 08:00:20 dist-test-slave-9bxp krb5kdc[14778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536020, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:20.707501 14900 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:37565
I20260430 08:00:20.707535 15030 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:37565 every 8 connection(s)
I20260430 08:00:20.708796 14900 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:00:20.709769 14875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:20.696007 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:47329 (local address 127.13.94.190:44423)
0430 08:00:20.696218 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:20.696225 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:20.697225 (+ 1000us) server_negotiation.cc:408] Connection header received
0430 08:00:20.698427 (+ 1202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:20.698431 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:20.698499 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:20.698601 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:20.700430 (+ 1829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.701105 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:20.701792 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.701991 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:20.705669 (+ 3678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:20.705693 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:20.705696 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:20.705728 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:20.707610 (+ 1882us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.708172 (+ 562us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.708176 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.708178 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.708242 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.708533 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.708536 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.708538 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.708790 (+ 252us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:20.709002 (+ 212us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:20.709462 (+ 460us) server_negotiation.cc:300] Negotiation successful
0430 08:00:20.709606 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 08:00:20.710500 14924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:20.696399 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:44423 (local address 127.13.94.129:47329)
0430 08:00:20.697032 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:20.697078 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:20.698033 (+ 955us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:20.698882 (+ 849us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:20.698894 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:20.699502 (+ 608us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:20.700249 (+ 747us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:20.700268 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.701255 (+ 987us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:20.701259 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:20.701659 (+ 400us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:20.701666 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.701914 (+ 248us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:20.702677 (+ 763us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:20.702703 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:20.705476 (+ 2773us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:20.707755 (+ 2279us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:20.707760 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:20.707770 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:20.708043 (+ 273us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:20.708341 (+ 298us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:20.708343 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:20.708345 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:20.708432 (+ 87us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:20.708925 (+ 493us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:20.708932 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:20.709189 (+ 257us) client_negotiation.cc:770] Sending connection context
0430 08:00:20.709407 (+ 218us) client_negotiation.cc:241] Negotiation successful
0430 08:00:20.709654 (+ 247us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":186,"threads_started":1}
I20260430 08:00:20.711862 15031 heartbeater.cc:344] Connected to a master server at 127.13.94.190:44423
I20260430 08:00:20.712029 15031 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:20.712632 15031 heartbeater.cc:507] Master 127.13.94.190:44423 requested a full tablet report, sending...
I20260430 08:00:20.714618 14825 ts_manager.cc:194] Registered new tserver with Master: e9b3cbe49cb1436d95c7a5f2087bcfcd (127.13.94.129:37565)
I20260430 08:00:20.715574 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 14900
I20260430 08:00:20.715665 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:20.715975 13690 external_mini_cluster.cc:1468] Setting key 5f96dc1707dd1aaf37a8bd3e28871402
I20260430 08:00:20.715996 14825 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:47329
I20260430 08:00:20.724442 14875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:20.714379 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:53957 (local address 127.13.94.190:44423)
0430 08:00:20.714505 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:20.714508 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:20.717429 (+ 2921us) server_negotiation.cc:408] Connection header received
0430 08:00:20.717588 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:20.717592 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:20.717643 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:20.717738 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:20.718337 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.718864 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:20.719615 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.719777 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:20.720818 (+ 1041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:20.720835 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:20.720837 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:20.720861 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:20.722443 (+ 1582us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.723072 (+ 629us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.723075 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.723077 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.723135 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.723456 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.723459 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.723461 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.723613 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:20.723699 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:20.724185 (+ 486us) server_negotiation.cc:300] Negotiation successful
0430 08:00:20.724298 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260430 08:00:20.724565 15037 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:20.714302 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:44423 (local address 127.13.94.129:53957)
0430 08:00:20.717326 (+ 3024us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:20.717355 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:20.717486 (+ 131us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:20.717756 (+ 270us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:20.717760 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:20.717980 (+ 220us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:20.718210 (+ 230us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:20.718216 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.718990 (+ 774us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:20.718994 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:20.719472 (+ 478us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:20.719483 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.719632 (+ 149us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:20.720266 (+ 634us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:20.720291 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:20.720628 (+ 337us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:20.722586 (+ 1958us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:20.722594 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:20.722600 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:20.722918 (+ 318us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:20.723252 (+ 334us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:20.723256 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:20.723261 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:20.723344 (+ 83us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:20.723717 (+ 373us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:20.723723 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:20.723842 (+ 119us) client_negotiation.cc:770] Sending connection context
0430 08:00:20.724120 (+ 278us) client_negotiation.cc:241] Negotiation successful
0430 08:00:20.724306 (+ 186us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2316,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:20.777014 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:44423
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:20.882750 15042 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:20.882994 15042 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:20.883095 15042 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:20.886523 15042 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:20.886590 15042 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:20.886684 15042 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 08:00:20.886758 15042 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:20.891206 15042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.94.190:44423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:20.892444 15042 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:20.893404 15042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:20.900601 15050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:20.900677 15042 server_base.cc:1061] running on GCE node
W20260430 08:00:20.900595 15048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:20.900601 15047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:20.901361 15042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:20.902127 15042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:20.903332 15042 hybrid_clock.cc:648] HybridClock initialized: now 1777536020903294 us; error 57 us; skew 500 ppm
Apr 30 08:00:20 dist-test-slave-9bxp krb5kdc[14778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536020, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:20.906203 15042 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:00:20.907303 15042 webserver.cc:492] Webserver started at http://127.13.94.130:41571/ using document root <none> and password file <none>
I20260430 08:00:20.907882 15042 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:20.907970 15042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:20.908198 15042 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:20.910014 15042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "4ffc1da8c78243918f943fc60c9c822f"
format_stamp: "Formatted at 2026-04-30 08:00:20 on dist-test-slave-9bxp"
server_key: "41343ada0fb59b716364081986e3e459"
server_key_iv: "aafff781879b034ff7de8d31c03c3c2a"
server_key_version: "encryptionkey@0"
I20260430 08:00:20.910516 15042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "4ffc1da8c78243918f943fc60c9c822f"
format_stamp: "Formatted at 2026-04-30 08:00:20 on dist-test-slave-9bxp"
server_key: "41343ada0fb59b716364081986e3e459"
server_key_iv: "aafff781879b034ff7de8d31c03c3c2a"
server_key_version: "encryptionkey@0"
I20260430 08:00:20.914170 15042 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260430 08:00:20.916430 15057 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:20.917546 15042 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:20.917690 15042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "4ffc1da8c78243918f943fc60c9c822f"
format_stamp: "Formatted at 2026-04-30 08:00:20 on dist-test-slave-9bxp"
server_key: "41343ada0fb59b716364081986e3e459"
server_key_iv: "aafff781879b034ff7de8d31c03c3c2a"
server_key_version: "encryptionkey@0"
I20260430 08:00:20.917809 15042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:20.931252 15042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:20.934386 15042 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:20.934618 15042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:20.936156 15042 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:20.936237 15042 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:20.936304 15042 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:20.936348 15042 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 08:00:20 dist-test-slave-9bxp krb5kdc[14778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536020, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:20.946815 15042 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:39225
I20260430 08:00:20.946866 15172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:39225 every 8 connection(s)
I20260430 08:00:20.948076 15042 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:00:20.950749 14875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:20.936909 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:50565 (local address 127.13.94.190:44423)
0430 08:00:20.937073 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:20.937078 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:20.938253 (+ 1175us) server_negotiation.cc:408] Connection header received
0430 08:00:20.939430 (+ 1177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:20.939436 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:20.939510 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:20.939615 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:20.941472 (+ 1857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.942339 (+ 867us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:20.943306 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.943520 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:20.947229 (+ 3709us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:20.947253 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:20.947255 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:20.947287 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:20.948941 (+ 1654us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.949566 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.949569 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.949571 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.949616 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.949868 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.949870 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.949871 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.950017 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:20.950087 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:20.950531 (+ 444us) server_negotiation.cc:300] Negotiation successful
0430 08:00:20.950625 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 08:00:20.951354 15066 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:20.937251 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:44423 (local address 127.13.94.130:50565)
0430 08:00:20.938086 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:20.938128 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:20.939155 (+ 1027us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:20.939826 (+ 671us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:20.939838 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:20.940455 (+ 617us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:20.941287 (+ 832us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:20.941305 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.942484 (+ 1179us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:20.942489 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:20.943166 (+ 677us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:20.943175 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.943415 (+ 240us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:20.945074 (+ 1659us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:20.945106 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:20.946958 (+ 1852us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:20.949208 (+ 2250us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:20.949214 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:20.949226 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:20.949456 (+ 230us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:20.949698 (+ 242us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:20.949701 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:20.949702 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:20.949785 (+ 83us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:20.950101 (+ 316us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:20.950105 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:20.950317 (+ 212us) client_negotiation.cc:770] Sending connection context
0430 08:00:20.950474 (+ 157us) client_negotiation.cc:241] Negotiation successful
0430 08:00:20.950661 (+ 187us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":523,"threads_started":1}
I20260430 08:00:20.952564 15173 heartbeater.cc:344] Connected to a master server at 127.13.94.190:44423
I20260430 08:00:20.952793 15173 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:20.953363 15173 heartbeater.cc:507] Master 127.13.94.190:44423 requested a full tablet report, sending...
I20260430 08:00:20.953511 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 15042
I20260430 08:00:20.953629 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:20.953866 13690 external_mini_cluster.cc:1468] Setting key 6b1e10f0259fb15b494e2233acc9ce73
I20260430 08:00:20.954519 14826 ts_manager.cc:194] Registered new tserver with Master: 4ffc1da8c78243918f943fc60c9c822f (127.13.94.130:39225)
I20260430 08:00:20.955233 14826 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:50565
I20260430 08:00:20.962533 14875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:20.955373 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:40137 (local address 127.13.94.190:44423)
0430 08:00:20.955502 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:20.955506 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:20.955698 (+ 192us) server_negotiation.cc:408] Connection header received
0430 08:00:20.955898 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:20.955905 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:20.955976 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:20.956067 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:20.956746 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.957323 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:20.957947 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.958168 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:20.959111 (+ 943us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:20.959138 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:20.959141 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:20.959165 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:20.960886 (+ 1721us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.961392 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.961399 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.961404 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.961462 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:20.961740 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:20.961743 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:20.961744 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:20.961917 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:20.962019 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:20.962303 (+ 284us) server_negotiation.cc:300] Negotiation successful
0430 08:00:20.962400 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260430 08:00:20.962651 15179 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:20.955011 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:44423 (local address 127.13.94.130:40137)
0430 08:00:20.955608 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:20.955626 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:20.955743 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:20.956102 (+ 359us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:20.956106 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:20.956315 (+ 209us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:20.956573 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:20.956581 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.957445 (+ 864us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:20.957449 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:20.957830 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:20.957837 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:20.957967 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:20.958618 (+ 651us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:20.958639 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:20.958985 (+ 346us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:20.961007 (+ 2022us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:20.961012 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:20.961015 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:20.961261 (+ 246us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:20.961563 (+ 302us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:20.961568 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:20.961573 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:20.961643 (+ 70us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:20.962019 (+ 376us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:20.962022 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:20.962132 (+ 110us) client_negotiation.cc:770] Sending connection context
0430 08:00:20.962252 (+ 120us) client_negotiation.cc:241] Negotiation successful
0430 08:00:20.962402 (+ 150us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:21.004971 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:44423
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:32773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:21.112418 15184 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:21.112761 15184 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:21.112890 15184 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:21.116304 15184 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:21.116436 15184 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:21.116536 15184 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 08:00:21.116629 15184 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:21.121102 15184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:32773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.94.190:44423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:21.122318 15184 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:21.123212 15184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:21.130100 15192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:21.130174 15190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:21.130095 15189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:21.130364 15184 server_base.cc:1061] running on GCE node
I20260430 08:00:21.130784 15184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:21.131354 15184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:21.132507 15184 hybrid_clock.cc:648] HybridClock initialized: now 1777536021132482 us; error 38 us; skew 500 ppm
Apr 30 08:00:21 dist-test-slave-9bxp krb5kdc[14778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:21.135252 15184 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:00:21.136299 15184 webserver.cc:492] Webserver started at http://127.13.94.131:40979/ using document root <none> and password file <none>
I20260430 08:00:21.136924 15184 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:21.136996 15184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:21.137214 15184 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:21.138914 15184 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "a60606e8041949829772e80d28768c9d"
format_stamp: "Formatted at 2026-04-30 08:00:21 on dist-test-slave-9bxp"
server_key: "dd2a500d4c77066d01e90c6832fa33d7"
server_key_iv: "86678a42cb8dec8f07bf00e00bad7d14"
server_key_version: "encryptionkey@0"
I20260430 08:00:21.139408 15184 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "a60606e8041949829772e80d28768c9d"
format_stamp: "Formatted at 2026-04-30 08:00:21 on dist-test-slave-9bxp"
server_key: "dd2a500d4c77066d01e90c6832fa33d7"
server_key_iv: "86678a42cb8dec8f07bf00e00bad7d14"
server_key_version: "encryptionkey@0"
I20260430 08:00:21.142813 15184 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:21.144945 15199 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:21.145992 15184 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:21.146168 15184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "a60606e8041949829772e80d28768c9d"
format_stamp: "Formatted at 2026-04-30 08:00:21 on dist-test-slave-9bxp"
server_key: "dd2a500d4c77066d01e90c6832fa33d7"
server_key_iv: "86678a42cb8dec8f07bf00e00bad7d14"
server_key_version: "encryptionkey@0"
I20260430 08:00:21.146288 15184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:21.165740 15184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:21.168452 15184 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:21.168654 15184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:21.170163 15184 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:21.170239 15184 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:21.170310 15184 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:21.170365 15184 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 08:00:21 dist-test-slave-9bxp krb5kdc[14778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536021, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:21.181022 15184 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:39687
I20260430 08:00:21.182022 15184 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:00:21.182518 15314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:39687 every 8 connection(s)
I20260430 08:00:21.184186 14875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.171054 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:44937 (local address 127.13.94.190:44423)
0430 08:00:21.171214 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.171218 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.171978 (+ 760us) server_negotiation.cc:408] Connection header received
0430 08:00:21.173030 (+ 1052us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.173040 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.173115 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.173209 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:21.175057 (+ 1848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.175950 (+ 893us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.176888 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.177134 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.180382 (+ 3248us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:21.180409 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:21.180412 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:21.180450 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:21.182184 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.182774 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.182778 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.182780 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.182843 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.183193 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.183196 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.183197 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.183356 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:21.183446 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.183924 (+ 478us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.184038 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260430 08:00:21.184784 15208 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.171411 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:44423 (local address 127.13.94.131:44937)
0430 08:00:21.171820 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.171860 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.172795 (+ 935us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.173659 (+ 864us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.173669 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.174190 (+ 521us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:21.174884 (+ 694us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.174899 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.176093 (+ 1194us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.176097 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.176748 (+ 651us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.176757 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.177001 (+ 244us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.177657 (+ 656us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:21.177684 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:21.180188 (+ 2504us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:21.182303 (+ 2115us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:21.182309 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:21.182322 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:21.182647 (+ 325us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:21.182929 (+ 282us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:21.182933 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:21.182935 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:21.183075 (+ 140us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:21.183455 (+ 380us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:21.183460 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:21.183658 (+ 198us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.183827 (+ 169us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.184054 (+ 227us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":126,"threads_started":1}
I20260430 08:00:21.186086 15315 heartbeater.cc:344] Connected to a master server at 127.13.94.190:44423
I20260430 08:00:21.186278 15315 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:21.186780 15315 heartbeater.cc:507] Master 127.13.94.190:44423 requested a full tablet report, sending...
I20260430 08:00:21.187875 14825 ts_manager.cc:194] Registered new tserver with Master: a60606e8041949829772e80d28768c9d (127.13.94.131:39687)
I20260430 08:00:21.188570 14825 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:44937
I20260430 08:00:21.191493 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 15184
I20260430 08:00:21.191617 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:21.191951 13690 external_mini_cluster.cc:1468] Setting key f7007a27665d2c472bc3264218d019fd
I20260430 08:00:21.194327 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 08:00:21.195371 15321 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.188351 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:44423 (local address 127.13.94.131:50835)
0430 08:00:21.188695 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.188733 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.188850 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.189155 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.189158 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.189370 (+ 212us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:21.189643 (+ 273us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.189653 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.190390 (+ 737us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.190394 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.190820 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.190827 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.190925 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.191494 (+ 569us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:21.191514 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:21.191912 (+ 398us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:21.193939 (+ 2027us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:21.193942 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:21.193945 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:21.194177 (+ 232us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:21.194482 (+ 305us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:21.194485 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:21.194487 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:21.194542 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:21.194927 (+ 385us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:21.194930 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:21.195005 (+ 75us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.195094 (+ 89us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.195217 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":118,"threads_started":1}
I20260430 08:00:21.195546 14875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.188465 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:50835 (local address 127.13.94.190:44423)
0430 08:00:21.188604 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.188608 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.188795 (+ 187us) server_negotiation.cc:408] Connection header received
0430 08:00:21.188956 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.188960 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.189008 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.189102 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:21.189760 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.190273 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.190925 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.191095 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.192121 (+ 1026us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:21.192145 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:21.192147 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:21.192172 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:21.193819 (+ 1647us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.194303 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.194307 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.194309 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.194368 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.194632 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.194635 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.194638 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.194812 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:21.194930 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.195183 (+ 253us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.195303 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Apr 30 08:00:21 dist-test-slave-9bxp krb5kdc[14778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536021, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 30 08:00:21 dist-test-slave-9bxp krb5kdc[14778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536021, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:21.219537 14875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.211742 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.13.94.190:44423)
0430 08:00:21.211901 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.211905 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.211918 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:00:21.211961 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.211964 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.212007 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.212101 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:21.213009 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.213506 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.214190 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.214408 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.216596 (+ 2188us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:21.216617 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:21.216619 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:21.216648 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:21.218008 (+ 1360us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.218421 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.218427 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.218433 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.218479 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.218721 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.218724 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.218726 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.218885 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:21.219036 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.219250 (+ 214us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.219360 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260430 08:00:21.222329 14825 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:00:21.224933 14825 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:21.238256 15332 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.233763 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:39687 (local address 127.0.0.1:56266)
0430 08:00:21.234104 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.234140 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.234281 (+ 141us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.235413 (+ 1132us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.235419 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.235441 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:21.235749 (+ 308us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.235756 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.236936 (+ 1180us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.236941 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.237757 (+ 816us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.237764 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.237871 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.237926 (+ 55us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.238018 (+ 92us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.238084 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":118,"threads_started":1}
I20260430 08:00:21.238463 15334 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.234266 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:39225 (local address 127.0.0.1:42790)
0430 08:00:21.234500 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.234523 (+ 23us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.234601 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.235684 (+ 1083us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.235687 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.235704 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:21.235930 (+ 226us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.235936 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.237305 (+ 1369us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.237309 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.238113 (+ 804us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.238120 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.238220 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.238231 (+ 11us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.238279 (+ 48us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.238343 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260430 08:00:21.238763 15335 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.234488 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:37565 (local address 127.0.0.1:37954)
0430 08:00:21.234751 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.234763 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.234834 (+ 71us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.235853 (+ 1019us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.235857 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.235876 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:21.236152 (+ 276us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.236159 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.237536 (+ 1377us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.237541 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.238392 (+ 851us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.238402 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.238503 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.238519 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.238574 (+ 55us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.238629 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260430 08:00:21.239212 15336 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.234632 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.13.94.130:39225)
0430 08:00:21.235259 (+ 627us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.235268 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.235289 (+ 21us) server_negotiation.cc:408] Connection header received
0430 08:00:21.235371 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.235376 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.235544 (+ 168us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.235691 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:21.236039 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.237187 (+ 1148us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.238266 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.238807 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.238939 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.239001 (+ 62us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.239073 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":98,"threads_started":1}
I20260430 08:00:21.239779 15333 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.233902 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.13.94.131:39687)
0430 08:00:21.234880 (+ 978us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.234889 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.234913 (+ 24us) server_negotiation.cc:408] Connection header received
0430 08:00:21.234993 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.234997 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.235205 (+ 208us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.235413 (+ 208us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:21.235874 (+ 461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.236818 (+ 944us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.238814 (+ 1996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.239409 (+ 595us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.239507 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.239565 (+ 58us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.239640 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":111,"threads_started":1}
I20260430 08:00:21.239809 15337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.235068 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.13.94.129:37565)
0430 08:00:21.235396 (+ 328us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.235401 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.235417 (+ 16us) server_negotiation.cc:408] Connection header received
0430 08:00:21.235484 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.235488 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.235607 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.235714 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:21.236287 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.237406 (+ 1119us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.238978 (+ 1572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.239497 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.239571 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.239637 (+ 66us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.239697 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
I20260430 08:00:21.242215 15249 tablet_service.cc:1511] Processing CreateTablet for tablet 035c0d98062d41b0bcaad848dbc77058 (DEFAULT_TABLE table=test-table [id=e87729e1b58c4c83acd27438c2870649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:21.242218 15107 tablet_service.cc:1511] Processing CreateTablet for tablet 035c0d98062d41b0bcaad848dbc77058 (DEFAULT_TABLE table=test-table [id=e87729e1b58c4c83acd27438c2870649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:21.242214 14965 tablet_service.cc:1511] Processing CreateTablet for tablet 035c0d98062d41b0bcaad848dbc77058 (DEFAULT_TABLE table=test-table [id=e87729e1b58c4c83acd27438c2870649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:21.243151 15249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 035c0d98062d41b0bcaad848dbc77058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:21.243151 14965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 035c0d98062d41b0bcaad848dbc77058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:21.243151 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 035c0d98062d41b0bcaad848dbc77058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:21.248600 15338 tablet_bootstrap.cc:492] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: Bootstrap starting.
I20260430 08:00:21.249963 15340 tablet_bootstrap.cc:492] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: Bootstrap starting.
I20260430 08:00:21.250053 15339 tablet_bootstrap.cc:492] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: Bootstrap starting.
I20260430 08:00:21.251511 15338 tablet_bootstrap.cc:654] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:21.252312 15340 tablet_bootstrap.cc:654] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:21.252312 15339 tablet_bootstrap.cc:654] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:21.252643 15338 log.cc:826] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:21.253227 15339 log.cc:826] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:21.253561 15340 log.cc:826] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:21.254607 15338 tablet_bootstrap.cc:492] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: No bootstrap required, opened a new log
I20260430 08:00:21.254801 15338 ts_tablet_manager.cc:1403] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260430 08:00:21.255097 15339 tablet_bootstrap.cc:492] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: No bootstrap required, opened a new log
I20260430 08:00:21.255295 15339 ts_tablet_manager.cc:1403] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 08:00:21.255494 15340 tablet_bootstrap.cc:492] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: No bootstrap required, opened a new log
I20260430 08:00:21.255664 15340 ts_tablet_manager.cc:1403] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260430 08:00:21.257895 15339 raft_consensus.cc:359] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } }
I20260430 08:00:21.258072 15339 raft_consensus.cc:385] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:21.258104 15339 raft_consensus.cc:740] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b3cbe49cb1436d95c7a5f2087bcfcd, State: Initialized, Role: FOLLOWER
I20260430 08:00:21.258555 15339 consensus_queue.cc:260] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } }
I20260430 08:00:21.258549 15340 raft_consensus.cc:359] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } }
I20260430 08:00:21.258494 15338 raft_consensus.cc:359] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } }
I20260430 08:00:21.258709 15340 raft_consensus.cc:385] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:21.258710 15338 raft_consensus.cc:385] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:21.258769 15338 raft_consensus.cc:740] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60606e8041949829772e80d28768c9d, State: Initialized, Role: FOLLOWER
I20260430 08:00:21.258810 15340 raft_consensus.cc:740] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ffc1da8c78243918f943fc60c9c822f, State: Initialized, Role: FOLLOWER
I20260430 08:00:21.259265 15340 consensus_queue.cc:260] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } }
I20260430 08:00:21.259420 15338 consensus_queue.cc:260] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } }
I20260430 08:00:21.259642 15339 ts_tablet_manager.cc:1434] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260430 08:00:21.260082 15031 heartbeater.cc:499] Master 127.13.94.190:44423 was elected leader, sending a full tablet report...
I20260430 08:00:21.260216 15340 ts_tablet_manager.cc:1434] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260430 08:00:21.260536 15173 heartbeater.cc:499] Master 127.13.94.190:44423 was elected leader, sending a full tablet report...
I20260430 08:00:21.260885 15315 heartbeater.cc:499] Master 127.13.94.190:44423 was elected leader, sending a full tablet report...
I20260430 08:00:21.261427 15338 ts_tablet_manager.cc:1434] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
W20260430 08:00:21.433024 15316 tablet.cc:2404] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:00:21.449745 15174 tablet.cc:2404] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:00:21.461215 15032 tablet.cc:2404] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:00:21.477960 15345 raft_consensus.cc:493] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:21.478193 15345 raft_consensus.cc:515] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } }
I20260430 08:00:21.479314 15345 leader_election.cc:290] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a60606e8041949829772e80d28768c9d (127.13.94.131:39687), e9b3cbe49cb1436d95c7a5f2087bcfcd (127.13.94.129:37565)
I20260430 08:00:21.483175 15348 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.480161 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:37565 (local address 127.13.94.130:59415)
0430 08:00:21.480469 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.480485 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.480572 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.480859 (+ 287us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.480861 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.480878 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:21.481166 (+ 288us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.481175 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.482092 (+ 917us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.482095 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.482740 (+ 645us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.482748 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.482851 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.482866 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.482913 (+ 47us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.482998 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":77,"threads_started":1}
I20260430 08:00:21.483345 15347 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.479676 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:39687 (local address 127.13.94.130:47489)
0430 08:00:21.479989 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.480003 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.480176 (+ 173us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.480558 (+ 382us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.480581 (+ 23us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.480615 (+ 34us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:21.480930 (+ 315us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.480939 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.481769 (+ 830us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.481773 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.482789 (+ 1016us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.482815 (+ 26us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.482947 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.482964 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.483015 (+ 51us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.483069 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":120,"threads_started":1}
I20260430 08:00:21.483783 15333 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.479759 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:47489 (local address 127.13.94.131:39687)
0430 08:00:21.479920 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.479926 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.480095 (+ 169us) server_negotiation.cc:408] Connection header received
0430 08:00:21.480365 (+ 270us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.480369 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.480429 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.480517 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:21.481059 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.481642 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.483003 (+ 1361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.483490 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.483533 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.483591 (+ 58us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.483657 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260430 08:00:21.484025 15337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.480303 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:59415 (local address 127.13.94.129:37565)
0430 08:00:21.480450 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.480454 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.480565 (+ 111us) server_negotiation.cc:408] Connection header received
0430 08:00:21.480673 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.480677 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.480758 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.480848 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:21.481335 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.481963 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.482862 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.483684 (+ 822us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.483716 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.483794 (+ 78us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.483861 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260430 08:00:21.484431 15269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "035c0d98062d41b0bcaad848dbc77058" candidate_uuid: "4ffc1da8c78243918f943fc60c9c822f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60606e8041949829772e80d28768c9d" is_pre_election: true
I20260430 08:00:21.484666 14985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "035c0d98062d41b0bcaad848dbc77058" candidate_uuid: "4ffc1da8c78243918f943fc60c9c822f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" is_pre_election: true
I20260430 08:00:21.484838 15269 raft_consensus.cc:2468] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ffc1da8c78243918f943fc60c9c822f in term 0.
I20260430 08:00:21.485128 14985 raft_consensus.cc:2468] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ffc1da8c78243918f943fc60c9c822f in term 0.
I20260430 08:00:21.485426 15060 leader_election.cc:304] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ffc1da8c78243918f943fc60c9c822f, a60606e8041949829772e80d28768c9d; no voters:
I20260430 08:00:21.485775 15345 raft_consensus.cc:2804] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:21.485872 15345 raft_consensus.cc:493] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:21.485917 15345 raft_consensus.cc:3060] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:21.487058 15345 raft_consensus.cc:515] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } }
I20260430 08:00:21.487514 15345 leader_election.cc:290] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [CANDIDATE]: Term 1 election: Requested vote from peers a60606e8041949829772e80d28768c9d (127.13.94.131:39687), e9b3cbe49cb1436d95c7a5f2087bcfcd (127.13.94.129:37565)
I20260430 08:00:21.488003 15269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "035c0d98062d41b0bcaad848dbc77058" candidate_uuid: "4ffc1da8c78243918f943fc60c9c822f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60606e8041949829772e80d28768c9d"
I20260430 08:00:21.488034 14985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "035c0d98062d41b0bcaad848dbc77058" candidate_uuid: "4ffc1da8c78243918f943fc60c9c822f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd"
I20260430 08:00:21.488160 15269 raft_consensus.cc:3060] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:21.488178 14985 raft_consensus.cc:3060] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:21.489296 14985 raft_consensus.cc:2468] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ffc1da8c78243918f943fc60c9c822f in term 1.
I20260430 08:00:21.489305 15269 raft_consensus.cc:2468] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ffc1da8c78243918f943fc60c9c822f in term 1.
I20260430 08:00:21.489660 15060 leader_election.cc:304] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ffc1da8c78243918f943fc60c9c822f, a60606e8041949829772e80d28768c9d; no voters:
I20260430 08:00:21.489840 15345 raft_consensus.cc:2804] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:21.490051 15345 raft_consensus.cc:697] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 1 LEADER]: Becoming Leader. State: Replica: 4ffc1da8c78243918f943fc60c9c822f, State: Running, Role: LEADER
I20260430 08:00:21.490394 15345 consensus_queue.cc:237] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } }
I20260430 08:00:21.493952 14826 catalog_manager.cc:5671] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ffc1da8c78243918f943fc60c9c822f (127.13.94.130). New cstate: current_term: 1 leader_uuid: "4ffc1da8c78243918f943fc60c9c822f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } health_report { overall_health: HEALTHY } } }
I20260430 08:00:21.537417 15336 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.534309 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.13.94.130:39225)
0430 08:00:21.534448 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.534452 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.534517 (+ 65us) server_negotiation.cc:408] Connection header received
0430 08:00:21.534692 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.534695 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.534748 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.534829 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:21.535167 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.535722 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.536412 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.536550 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.536595 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:21.537036 (+ 441us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:21.537120 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.537255 (+ 135us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.537302 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 08:00:21.542937 15269 raft_consensus.cc:1275] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Refusing update from remote peer 4ffc1da8c78243918f943fc60c9c822f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:21.542937 14985 raft_consensus.cc:1275] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 FOLLOWER]: Refusing update from remote peer 4ffc1da8c78243918f943fc60c9c822f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:21.543614 15345 consensus_queue.cc:1048] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:21.543790 15349 consensus_queue.cc:1048] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:21.551381 15352 mvcc.cc:204] Tried to move back new op lower bound from 7280787544232906752 to 7280787544027783168. Current Snapshot: MvccSnapshot[applied={T|T < 7280787544232906752}]
I20260430 08:00:21.551633 15354 mvcc.cc:204] Tried to move back new op lower bound from 7280787544232906752 to 7280787544027783168. Current Snapshot: MvccSnapshot[applied={T|T < 7280787544232906752}]
I20260430 08:00:21.554082 15353 mvcc.cc:204] Tried to move back new op lower bound from 7280787544232906752 to 7280787544027783168. Current Snapshot: MvccSnapshot[applied={T|T < 7280787544232906752}]
I20260430 08:00:21.563208 14825 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:21.563400 14825 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:21.565935 14825 catalog_manager.cc:5958] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60: Sending DeleteTablet for 3 replicas of tablet 035c0d98062d41b0bcaad848dbc77058
I20260430 08:00:21.566887 15107 tablet_service.cc:1558] Processing DeleteTablet for tablet 035c0d98062d41b0bcaad848dbc77058 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:21 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42790
I20260430 08:00:21.567055 15249 tablet_service.cc:1558] Processing DeleteTablet for tablet 035c0d98062d41b0bcaad848dbc77058 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:21 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:56266
I20260430 08:00:21.567520 14965 tablet_service.cc:1558] Processing DeleteTablet for tablet 035c0d98062d41b0bcaad848dbc77058 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:21 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:37954
I20260430 08:00:21.567597 15362 tablet_replica.cc:333] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: stopping tablet replica
I20260430 08:00:21.567700 14825 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 08:00:21.567981 15364 tablet_replica.cc:333] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: stopping tablet replica
W20260430 08:00:21.568161 14825 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:21.568243 15364 raft_consensus.cc:2243] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:21.568176 15362 raft_consensus.cc:2243] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:00:21.568573 15364 raft_consensus.cc:2272] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:21.568943 15362 raft_consensus.cc:2272] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:21.568989 15363 tablet_replica.cc:333] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: stopping tablet replica
I20260430 08:00:21.570431 15362 ts_tablet_manager.cc:1916] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:21.570441 15364 ts_tablet_manager.cc:1916] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:21.573011 15362 ts_tablet_manager.cc:1929] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:00:21.573093 15364 ts_tablet_manager.cc:1929] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:00:21.573215 15364 log.cc:1199] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/035c0d98062d41b0bcaad848dbc77058
I20260430 08:00:21.573167 15362 log.cc:1199] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/035c0d98062d41b0bcaad848dbc77058
I20260430 08:00:21.573566 15364 ts_tablet_manager.cc:1950] T 035c0d98062d41b0bcaad848dbc77058 P e9b3cbe49cb1436d95c7a5f2087bcfcd: Deleting consensus metadata
I20260430 08:00:21.573794 15362 ts_tablet_manager.cc:1950] T 035c0d98062d41b0bcaad848dbc77058 P 4ffc1da8c78243918f943fc60c9c822f: Deleting consensus metadata
I20260430 08:00:21.574456 15249 tablet_service.cc:1511] Processing CreateTablet for tablet c1287eb0101a4431bebe9f793567419a (DEFAULT_TABLE table=test-table [id=4642952056f847f98b1c672b7f22e269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:21.574520 14965 tablet_service.cc:1511] Processing CreateTablet for tablet c1287eb0101a4431bebe9f793567419a (DEFAULT_TABLE table=test-table [id=4642952056f847f98b1c672b7f22e269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:21.574738 15249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1287eb0101a4431bebe9f793567419a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:21.574770 14965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1287eb0101a4431bebe9f793567419a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:21.576498 14811 catalog_manager.cc:5002] TS 4ffc1da8c78243918f943fc60c9c822f (127.13.94.130:39225): tablet 035c0d98062d41b0bcaad848dbc77058 (table test-table [id=e87729e1b58c4c83acd27438c2870649]) successfully deleted
I20260430 08:00:21.577041 15339 tablet_bootstrap.cc:492] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: Bootstrap starting.
I20260430 08:00:21.577068 15338 tablet_bootstrap.cc:492] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: Bootstrap starting.
I20260430 08:00:21.574070 15107 tablet_service.cc:1511] Processing CreateTablet for tablet c1287eb0101a4431bebe9f793567419a (DEFAULT_TABLE table=test-table [id=4642952056f847f98b1c672b7f22e269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:21.578025 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1287eb0101a4431bebe9f793567419a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:21.578068 15338 tablet_bootstrap.cc:654] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:21.578089 15339 tablet_bootstrap.cc:654] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:21.579764 15363 raft_consensus.cc:2243] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:21.580323 15340 tablet_bootstrap.cc:492] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: Bootstrap starting.
I20260430 08:00:21.580969 15363 raft_consensus.cc:2272] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:21.582667 15340 tablet_bootstrap.cc:654] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:21.582646 15338 tablet_bootstrap.cc:492] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: No bootstrap required, opened a new log
I20260430 08:00:21.582779 15338 ts_tablet_manager.cc:1403] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260430 08:00:21.583586 15338 raft_consensus.cc:359] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.583942 15338 raft_consensus.cc:385] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:21.584003 15338 raft_consensus.cc:740] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60606e8041949829772e80d28768c9d, State: Initialized, Role: FOLLOWER
I20260430 08:00:21.584132 15339 tablet_bootstrap.cc:492] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: No bootstrap required, opened a new log
I20260430 08:00:21.584223 15339 ts_tablet_manager.cc:1403] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260430 08:00:21.584259 15338 consensus_queue.cc:260] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.584551 15338 ts_tablet_manager.cc:1434] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260430 08:00:21.584774 15339 raft_consensus.cc:359] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.584877 15339 raft_consensus.cc:385] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:21.584916 15339 raft_consensus.cc:740] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b3cbe49cb1436d95c7a5f2087bcfcd, State: Initialized, Role: FOLLOWER
I20260430 08:00:21.585045 15339 consensus_queue.cc:260] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.585145 14812 catalog_manager.cc:5002] TS e9b3cbe49cb1436d95c7a5f2087bcfcd (127.13.94.129:37565): tablet 035c0d98062d41b0bcaad848dbc77058 (table test-table [id=e87729e1b58c4c83acd27438c2870649]) successfully deleted
I20260430 08:00:21.585301 15339 ts_tablet_manager.cc:1434] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:21.586715 15340 tablet_bootstrap.cc:492] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: No bootstrap required, opened a new log
I20260430 08:00:21.586802 15340 ts_tablet_manager.cc:1403] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260430 08:00:21.587244 15340 raft_consensus.cc:359] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.587348 15340 raft_consensus.cc:385] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:21.587379 15340 raft_consensus.cc:740] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ffc1da8c78243918f943fc60c9c822f, State: Initialized, Role: FOLLOWER
I20260430 08:00:21.587496 15340 consensus_queue.cc:260] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.587734 15340 ts_tablet_manager.cc:1434] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260430 08:00:21.589388 15363 ts_tablet_manager.cc:1916] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:21.591938 15363 ts_tablet_manager.cc:1929] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:00:21.592082 15363 log.cc:1199] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/035c0d98062d41b0bcaad848dbc77058
I20260430 08:00:21.592406 15363 ts_tablet_manager.cc:1950] T 035c0d98062d41b0bcaad848dbc77058 P a60606e8041949829772e80d28768c9d: Deleting consensus metadata
I20260430 08:00:21.593695 14812 catalog_manager.cc:5002] TS a60606e8041949829772e80d28768c9d (127.13.94.131:39687): tablet 035c0d98062d41b0bcaad848dbc77058 (table test-table [id=e87729e1b58c4c83acd27438c2870649]) successfully deleted
I20260430 08:00:21.709546 15344 raft_consensus.cc:493] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:21.709708 15344 raft_consensus.cc:515] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.711046 15344 leader_election.cc:290] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ffc1da8c78243918f943fc60c9c822f (127.13.94.130:39225), a60606e8041949829772e80d28768c9d (127.13.94.131:39687)
I20260430 08:00:21.715576 15336 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.711591 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:37225 (local address 127.13.94.130:39225)
0430 08:00:21.711711 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.711715 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.711913 (+ 198us) server_negotiation.cc:408] Connection header received
0430 08:00:21.712091 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.712095 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.712152 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.712197 (+ 45us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:21.712799 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.713555 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.714706 (+ 1151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.715172 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.715210 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.715401 (+ 191us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.715457 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260430 08:00:21.716099 15368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.711222 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:39225 (local address 127.13.94.129:37225)
0430 08:00:21.711838 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.711852 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.711982 (+ 130us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.712319 (+ 337us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.712322 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.712337 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:21.712661 (+ 324us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.712667 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.713686 (+ 1019us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.713689 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.714569 (+ 880us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.714577 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.715268 (+ 691us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.715281 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.715919 (+ 638us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.715978 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":65,"threads_started":1}
I20260430 08:00:21.716311 15333 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.711834 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:43311 (local address 127.13.94.131:39687)
0430 08:00:21.711976 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.711982 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.712255 (+ 273us) server_negotiation.cc:408] Connection header received
0430 08:00:21.712481 (+ 226us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.712486 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.712547 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.712633 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:21.713449 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.714144 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.715340 (+ 1196us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.715806 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.715843 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.716087 (+ 244us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.716172 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 08:00:21.716754 15369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.711423 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:39687 (local address 127.13.94.129:43311)
0430 08:00:21.712158 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.712174 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.712337 (+ 163us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.712956 (+ 619us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.712959 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.712984 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:21.713277 (+ 293us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.713288 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.714308 (+ 1020us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.714312 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.715176 (+ 864us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.715187 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.715918 (+ 731us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.715933 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.716521 (+ 588us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.716582 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":111,"threads_started":1}
I20260430 08:00:21.716805 15127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1287eb0101a4431bebe9f793567419a" candidate_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ffc1da8c78243918f943fc60c9c822f" is_pre_election: true
I20260430 08:00:21.717010 15127 raft_consensus.cc:2468] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b3cbe49cb1436d95c7a5f2087bcfcd in term 0.
I20260430 08:00:21.717223 15269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1287eb0101a4431bebe9f793567419a" candidate_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60606e8041949829772e80d28768c9d" is_pre_election: true
I20260430 08:00:21.717376 15269 raft_consensus.cc:2468] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b3cbe49cb1436d95c7a5f2087bcfcd in term 0.
I20260430 08:00:21.717664 14917 leader_election.cc:304] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ffc1da8c78243918f943fc60c9c822f, e9b3cbe49cb1436d95c7a5f2087bcfcd; no voters:
I20260430 08:00:21.717955 15344 raft_consensus.cc:2804] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:21.718030 15344 raft_consensus.cc:493] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:21.718083 15344 raft_consensus.cc:3060] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:21.719372 15344 raft_consensus.cc:515] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.719851 15344 leader_election.cc:290] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [CANDIDATE]: Term 1 election: Requested vote from peers 4ffc1da8c78243918f943fc60c9c822f (127.13.94.130:39225), a60606e8041949829772e80d28768c9d (127.13.94.131:39687)
I20260430 08:00:21.720355 15269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1287eb0101a4431bebe9f793567419a" candidate_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60606e8041949829772e80d28768c9d"
I20260430 08:00:21.720489 15269 raft_consensus.cc:3060] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:21.720448 15127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1287eb0101a4431bebe9f793567419a" candidate_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ffc1da8c78243918f943fc60c9c822f"
I20260430 08:00:21.720772 15127 raft_consensus.cc:3060] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:21.721774 15269 raft_consensus.cc:2468] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b3cbe49cb1436d95c7a5f2087bcfcd in term 1.
I20260430 08:00:21.722121 15127 raft_consensus.cc:2468] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b3cbe49cb1436d95c7a5f2087bcfcd in term 1.
I20260430 08:00:21.722182 14918 leader_election.cc:304] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60606e8041949829772e80d28768c9d, e9b3cbe49cb1436d95c7a5f2087bcfcd; no voters:
I20260430 08:00:21.722419 15344 raft_consensus.cc:2804] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:21.722636 15344 raft_consensus.cc:697] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 LEADER]: Becoming Leader. State: Replica: e9b3cbe49cb1436d95c7a5f2087bcfcd, State: Running, Role: LEADER
I20260430 08:00:21.722910 15344 consensus_queue.cc:237] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.725490 14825 catalog_manager.cc:5671] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd reported cstate change: term changed from 0 to 1, leader changed from <none> to e9b3cbe49cb1436d95c7a5f2087bcfcd (127.13.94.129). New cstate: current_term: 1 leader_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } health_report { overall_health: UNKNOWN } } }
Apr 30 08:00:21 dist-test-slave-9bxp krb5kdc[14778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536020, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:21.774338 15378 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.762675 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.13.94.190:44423)
0430 08:00:21.762975 (+ 300us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.762980 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.763000 (+ 20us) server_negotiation.cc:408] Connection header received
0430 08:00:21.763109 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.763119 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.763176 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.763282 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:21.764067 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.764985 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.765610 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.765812 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.771328 (+ 5516us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:21.771356 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:21.771358 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:21.771380 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:21.772665 (+ 1285us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.773182 (+ 517us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.773185 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.773188 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.773236 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.773466 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.773471 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.773473 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.773621 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:21.773722 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.774009 (+ 287us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.774212 (+ 203us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":219,"thread_start_us":111,"threads_started":1}
I20260430 08:00:21.774338 15377 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.762609 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:44423 (local address 127.0.0.1:44214)
0430 08:00:21.762881 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.762898 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.763007 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.763324 (+ 317us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.763326 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.763753 (+ 427us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:21.763947 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.763953 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.765101 (+ 1148us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.765104 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.765502 (+ 398us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.765509 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.765603 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.769603 (+ 4000us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:21.769618 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:21.771175 (+ 1557us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:21.772844 (+ 1669us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:21.772853 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:21.772857 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:21.773076 (+ 219us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:21.773333 (+ 257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:21.773340 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:21.773341 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:21.773383 (+ 42us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:21.773739 (+ 356us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:21.773745 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:21.773839 (+ 94us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.773963 (+ 124us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.774136 (+ 173us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260430 08:00:21.778456 14825 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:44214:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260430 08:00:21.779095 14825 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:21.784025 14965 tablet_service.cc:1511] Processing CreateTablet for tablet a1934e280c2c485fb4af2c022d40f65d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6eb338216fa456d94d2097365d8dace]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 08:00:21.784476 14965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1934e280c2c485fb4af2c022d40f65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:21.784399 15107 tablet_service.cc:1511] Processing CreateTablet for tablet a1934e280c2c485fb4af2c022d40f65d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6eb338216fa456d94d2097365d8dace]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 08:00:21.784513 15249 tablet_service.cc:1511] Processing CreateTablet for tablet a1934e280c2c485fb4af2c022d40f65d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6eb338216fa456d94d2097365d8dace]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 08:00:21.784795 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1934e280c2c485fb4af2c022d40f65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:21.784999 15249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1934e280c2c485fb4af2c022d40f65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:21.787318 15340 tablet_bootstrap.cc:492] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: Bootstrap starting.
I20260430 08:00:21.787362 15338 tablet_bootstrap.cc:492] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: Bootstrap starting.
I20260430 08:00:21.787504 15339 tablet_bootstrap.cc:492] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: Bootstrap starting.
I20260430 08:00:21.788241 15338 tablet_bootstrap.cc:654] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:21.788288 15340 tablet_bootstrap.cc:654] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:21.788440 15339 tablet_bootstrap.cc:654] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:21.789688 15338 tablet_bootstrap.cc:492] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: No bootstrap required, opened a new log
I20260430 08:00:21.789779 15338 ts_tablet_manager.cc:1403] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:21.789821 15340 tablet_bootstrap.cc:492] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: No bootstrap required, opened a new log
I20260430 08:00:21.789884 15339 tablet_bootstrap.cc:492] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: No bootstrap required, opened a new log
I20260430 08:00:21.789897 15340 ts_tablet_manager.cc:1403] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 08:00:21.789966 15339 ts_tablet_manager.cc:1403] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 08:00:21.790302 15338 raft_consensus.cc:359] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.790323 15340 raft_consensus.cc:359] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.790369 15339 raft_consensus.cc:359] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.790423 15338 raft_consensus.cc:385] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:21.790436 15340 raft_consensus.cc:385] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:21.790453 15339 raft_consensus.cc:385] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:21.790467 15338 raft_consensus.cc:740] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60606e8041949829772e80d28768c9d, State: Initialized, Role: FOLLOWER
I20260430 08:00:21.790474 15340 raft_consensus.cc:740] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ffc1da8c78243918f943fc60c9c822f, State: Initialized, Role: FOLLOWER
I20260430 08:00:21.790490 15339 raft_consensus.cc:740] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b3cbe49cb1436d95c7a5f2087bcfcd, State: Initialized, Role: FOLLOWER
I20260430 08:00:21.790587 15338 consensus_queue.cc:260] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.790601 15340 consensus_queue.cc:260] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.790601 15339 consensus_queue.cc:260] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.790879 15340 ts_tablet_manager.cc:1434] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:21.790870 15344 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.791018 15344 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:21.790900 15339 ts_tablet_manager.cc:1434] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:21.791110 15359 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.791195 15359 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:21.791365 15346 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.791514 15346 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:21.792110 15338 ts_tablet_manager.cc:1434] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:21.845643 15344 raft_consensus.cc:493] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:21.845824 15344 raft_consensus.cc:515] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.846302 15344 leader_election.cc:290] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ffc1da8c78243918f943fc60c9c822f (127.13.94.130:39225), a60606e8041949829772e80d28768c9d (127.13.94.131:39687)
I20260430 08:00:21.846913 15269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1934e280c2c485fb4af2c022d40f65d" candidate_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60606e8041949829772e80d28768c9d" is_pre_election: true
I20260430 08:00:21.846913 15127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1934e280c2c485fb4af2c022d40f65d" candidate_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ffc1da8c78243918f943fc60c9c822f" is_pre_election: true
I20260430 08:00:21.847069 15127 raft_consensus.cc:2468] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b3cbe49cb1436d95c7a5f2087bcfcd in term 0.
I20260430 08:00:21.847069 15269 raft_consensus.cc:2468] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b3cbe49cb1436d95c7a5f2087bcfcd in term 0.
I20260430 08:00:21.847391 14918 leader_election.cc:304] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a60606e8041949829772e80d28768c9d, e9b3cbe49cb1436d95c7a5f2087bcfcd; no voters:
I20260430 08:00:21.847615 15344 raft_consensus.cc:2804] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:21.847687 15344 raft_consensus.cc:493] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:21.847745 15344 raft_consensus.cc:3060] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:21.848825 15344 raft_consensus.cc:515] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.849196 15344 leader_election.cc:290] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [CANDIDATE]: Term 1 election: Requested vote from peers 4ffc1da8c78243918f943fc60c9c822f (127.13.94.130:39225), a60606e8041949829772e80d28768c9d (127.13.94.131:39687)
I20260430 08:00:21.849624 15269 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1934e280c2c485fb4af2c022d40f65d" candidate_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a60606e8041949829772e80d28768c9d"
I20260430 08:00:21.849632 15127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1934e280c2c485fb4af2c022d40f65d" candidate_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ffc1da8c78243918f943fc60c9c822f"
I20260430 08:00:21.849767 15269 raft_consensus.cc:3060] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:21.849767 15127 raft_consensus.cc:3060] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:21.850895 15127 raft_consensus.cc:2468] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b3cbe49cb1436d95c7a5f2087bcfcd in term 1.
I20260430 08:00:21.850895 15269 raft_consensus.cc:2468] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b3cbe49cb1436d95c7a5f2087bcfcd in term 1.
I20260430 08:00:21.851200 14917 leader_election.cc:304] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ffc1da8c78243918f943fc60c9c822f, e9b3cbe49cb1436d95c7a5f2087bcfcd; no voters:
I20260430 08:00:21.851388 15344 raft_consensus.cc:2804] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:21.851509 15344 raft_consensus.cc:697] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 LEADER]: Becoming Leader. State: Replica: e9b3cbe49cb1436d95c7a5f2087bcfcd, State: Running, Role: LEADER
I20260430 08:00:21.851651 15344 consensus_queue.cc:237] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } }
I20260430 08:00:21.852448 15370 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: TxnStatusTablet state changed. Reason: New leader e9b3cbe49cb1436d95c7a5f2087bcfcd. Latest consensus state: current_term: 1 leader_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.852547 15370 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: This TxnStatusTablet replica's current role is: LEADER
I20260430 08:00:21.852878 15383 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260430 08:00:21.853070 15383 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260430 08:00:21.853708 14825 catalog_manager.cc:5671] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd reported cstate change: term changed from 0 to 1, leader changed from <none> to e9b3cbe49cb1436d95c7a5f2087bcfcd (127.13.94.129). New cstate: current_term: 1 leader_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } health_report { overall_health: UNKNOWN } } }
I20260430 08:00:21.873260 15337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.869707 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.13.94.129:37565)
0430 08:00:21.869836 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.869839 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.869853 (+ 14us) server_negotiation.cc:408] Connection header received
0430 08:00:21.870032 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.870035 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.870089 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.870168 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:21.870599 (+ 431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.871243 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.872265 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.872406 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.872452 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:21.872887 (+ 435us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:21.872979 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.873094 (+ 115us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.873142 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260430 08:00:21.873301 15377 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.869559 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:37565 (local address 127.0.0.1:37968)
0430 08:00:21.869750 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.869766 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.869894 (+ 128us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.870201 (+ 307us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.870204 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.870213 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:00:21.870452 (+ 239us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.870459 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.871415 (+ 956us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.871420 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.872095 (+ 675us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.872105 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.872260 (+ 155us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.872306 (+ 46us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:00:21.872987 (+ 681us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:00:21.872999 (+ 12us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.873076 (+ 77us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.873159 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260430 08:00:21.876430 15269 raft_consensus.cc:1275] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Refusing update from remote peer e9b3cbe49cb1436d95c7a5f2087bcfcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:21.876633 15127 raft_consensus.cc:1275] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f [term 1 FOLLOWER]: Refusing update from remote peer e9b3cbe49cb1436d95c7a5f2087bcfcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:21.877064 15344 consensus_queue.cc:1048] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:21.877228 15370 consensus_queue.cc:1048] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:21.878851 15346 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: TxnStatusTablet state changed. Reason: New leader e9b3cbe49cb1436d95c7a5f2087bcfcd. Latest consensus state: current_term: 1 leader_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.878968 15346 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:21.879832 15359 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: TxnStatusTablet state changed. Reason: New leader e9b3cbe49cb1436d95c7a5f2087bcfcd. Latest consensus state: current_term: 1 leader_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.879935 15359 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:21.881745 15344 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.881870 15344 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: This TxnStatusTablet replica's current role is: LEADER
I20260430 08:00:21.882134 15388 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.882259 15388 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P e9b3cbe49cb1436d95c7a5f2087bcfcd: This TxnStatusTablet replica's current role is: LEADER
I20260430 08:00:21.884094 15359 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.884212 15359 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P 4ffc1da8c78243918f943fc60c9c822f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:21.885777 15346 tablet_replica.cc:442] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9b3cbe49cb1436d95c7a5f2087bcfcd" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37565 } } peers { permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 } } peers { permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 } } }
I20260430 08:00:21.885896 15346 tablet_replica.cc:445] T a1934e280c2c485fb4af2c022d40f65d P a60606e8041949829772e80d28768c9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:21.890692 15337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.887644 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.13.94.129:37565)
0430 08:00:21.887756 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.887760 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.887959 (+ 199us) server_negotiation.cc:408] Connection header received
0430 08:00:21.888164 (+ 205us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.888168 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.888224 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.888314 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:21.888680 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.889217 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.889930 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.890156 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.890198 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:21.890266 (+ 68us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:21.890361 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.890497 (+ 136us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.890545 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260430 08:00:21.901705 15337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.898326 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:47455 (local address 127.13.94.129:37565)
0430 08:00:21.898466 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.898470 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.898707 (+ 237us) server_negotiation.cc:408] Connection header received
0430 08:00:21.898862 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.898865 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.898945 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.899016 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:21.899508 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.900033 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.901055 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.901193 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.901231 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:21.901297 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:21.901367 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.901530 (+ 163us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.901570 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 08:00:21.901721 15391 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.898244 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:37565 (local address 127.13.94.129:47455)
0430 08:00:21.898617 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:21.898631 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:21.898742 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:21.899054 (+ 312us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:21.899057 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:21.899066 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:00:21.899390 (+ 324us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.899397 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.900187 (+ 790us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.900191 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:21.900901 (+ 710us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:21.900910 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.901053 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.901101 (+ 48us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:00:21.901400 (+ 299us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:00:21.901406 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 08:00:21.901477 (+ 71us) client_negotiation.cc:241] Negotiation successful
0430 08:00:21.901543 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":103,"threads_started":1}
I20260430 08:00:21.908512 15269 raft_consensus.cc:1275] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Refusing update from remote peer e9b3cbe49cb1436d95c7a5f2087bcfcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:21.908690 15127 raft_consensus.cc:1275] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [term 1 FOLLOWER]: Refusing update from remote peer e9b3cbe49cb1436d95c7a5f2087bcfcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:21.909000 15388 consensus_queue.cc:1048] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a60606e8041949829772e80d28768c9d" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 39687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:21.909149 15344 consensus_queue.cc:1048] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ffc1da8c78243918f943fc60c9c822f" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:21.949894 14825 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:21.950068 14825 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:21.952436 14825 catalog_manager.cc:5958] T 00000000000000000000000000000000 P c45804c8211f45679698e690f3c71d60: Sending DeleteTablet for 3 replicas of tablet c1287eb0101a4431bebe9f793567419a
I20260430 08:00:21.953126 14964 tablet_service.cc:1558] Processing DeleteTablet for tablet c1287eb0101a4431bebe9f793567419a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:21 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:37954
I20260430 08:00:21.953264 15249 tablet_service.cc:1558] Processing DeleteTablet for tablet c1287eb0101a4431bebe9f793567419a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:21 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:56266
I20260430 08:00:21.953337 15364 tablet_replica.cc:333] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: stopping tablet replica
I20260430 08:00:21.953459 15364 raft_consensus.cc:2243] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:00:21.953473 15363 tablet_replica.cc:333] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: stopping tablet replica
I20260430 08:00:21.953584 15363 raft_consensus.cc:2243] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:21.953708 15363 raft_consensus.cc:2272] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:21.953773 15364 raft_consensus.cc:2272] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:21.954011 15107 tablet_service.cc:1558] Processing DeleteTablet for tablet c1287eb0101a4431bebe9f793567419a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:21 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42790
I20260430 08:00:21.954188 15362 tablet_replica.cc:333] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: stopping tablet replica
I20260430 08:00:21.954293 15362 raft_consensus.cc:2243] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:21.954396 15362 raft_consensus.cc:2272] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:21.954571 15364 ts_tablet_manager.cc:1916] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:21.955310 15362 ts_tablet_manager.cc:1916] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:21.957361 15364 ts_tablet_manager.cc:1929] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:00:21.957424 15364 log.cc:1199] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/c1287eb0101a4431bebe9f793567419a
I20260430 08:00:21.957674 15364 ts_tablet_manager.cc:1950] T c1287eb0101a4431bebe9f793567419a P e9b3cbe49cb1436d95c7a5f2087bcfcd: Deleting consensus metadata
I20260430 08:00:21.957963 15362 ts_tablet_manager.cc:1929] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:00:21.958026 15362 log.cc:1199] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/c1287eb0101a4431bebe9f793567419a
I20260430 08:00:21.958261 15362 ts_tablet_manager.cc:1950] T c1287eb0101a4431bebe9f793567419a P 4ffc1da8c78243918f943fc60c9c822f: Deleting consensus metadata
I20260430 08:00:21.958478 14812 catalog_manager.cc:5002] TS e9b3cbe49cb1436d95c7a5f2087bcfcd (127.13.94.129:37565): tablet c1287eb0101a4431bebe9f793567419a (table test-table [id=4642952056f847f98b1c672b7f22e269]) successfully deleted
I20260430 08:00:21.959478 15363 ts_tablet_manager.cc:1916] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: Deleting tablet data with delete state TABLET_DATA_DELETED
Apr 30 08:00:21 dist-test-slave-9bxp krb5kdc[14778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536021, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:00:21.961016 14811 catalog_manager.cc:5002] TS 4ffc1da8c78243918f943fc60c9c822f (127.13.94.130:39225): tablet c1287eb0101a4431bebe9f793567419a (table test-table [id=4642952056f847f98b1c672b7f22e269]) successfully deleted
I20260430 08:00:21.962638 15363 ts_tablet_manager.cc:1929] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:00:21.963292 15363 log.cc:1199] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/c1287eb0101a4431bebe9f793567419a
I20260430 08:00:21.963634 15363 ts_tablet_manager.cc:1950] T c1287eb0101a4431bebe9f793567419a P a60606e8041949829772e80d28768c9d: Deleting consensus metadata
I20260430 08:00:21.964509 14812 catalog_manager.cc:5002] TS a60606e8041949829772e80d28768c9d (127.13.94.131:39687): tablet c1287eb0101a4431bebe9f793567419a (table test-table [id=4642952056f847f98b1c672b7f22e269]) successfully deleted
I20260430 08:00:21.965507 15337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.954648 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37974 (local address 127.13.94.129:37565)
0430 08:00:21.954759 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.954763 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.955269 (+ 506us) server_negotiation.cc:408] Connection header received
0430 08:00:21.955455 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.955458 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.955504 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.955552 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:21.956635 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.957388 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.958254 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.958449 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.961380 (+ 2931us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:21.961405 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:21.961419 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:21.961449 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:21.963631 (+ 2182us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.964138 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.964145 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.964148 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.964202 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.964457 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.964461 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.964462 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.964660 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:21.964846 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.965204 (+ 358us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.965338 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260430 08:00:21.966410 15013 server_base.cc:1143] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37974
I20260430 08:00:21.981207 15378 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:21.968647 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.13.94.190:44423)
0430 08:00:21.968815 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:21.968819 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:21.973127 (+ 4308us) server_negotiation.cc:408] Connection header received
0430 08:00:21.973263 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:21.973266 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:21.973333 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:21.973430 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:21.974518 (+ 1088us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.975365 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:21.976341 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:21.976496 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:21.977928 (+ 1432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:21.977956 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:21.977958 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:21.978021 (+ 63us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:21.979520 (+ 1499us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.980009 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.980011 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.980013 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.980057 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:21.980286 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:21.980288 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:21.980290 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:21.980435 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:21.980535 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:21.980852 (+ 317us) server_negotiation.cc:300] Negotiation successful
0430 08:00:21.980962 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260430 08:00:21.981684 14825 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44218
I20260430 08:00:21.983060 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 14900
I20260430 08:00:21.990206 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 15042
I20260430 08:00:21.997272 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 15184
I20260430 08:00:22.003602 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 14794
2026-04-30T08:00:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3962 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:22 dist-test-slave-9bxp krb5kdc[15401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:22 dist-test-slave-9bxp krb5kdc[15401](info): set up 2 sockets
Apr 30 08:00:22 dist-test-slave-9bxp krb5kdc[15401](info): commencing operation
krb5kdc: starting...
W20260430 08:00:24.021224 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.994s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:00:24 dist-test-slave-9bxp krb5kdc[15401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:24.161890 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:46039
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:46039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:24.264997 15417 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:24.265295 15417 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:24.265403 15417 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:24.268770 15417 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:24.268877 15417 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260430 08:00:24.268913 15417 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:24.268960 15417 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:24.269002 15417 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:24.269057 15417 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:24.273587 15417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:46039
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:46039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:24.274694 15417 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:24.275555 15417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:24.281309 15425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:24.281309 15422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:24.281309 15423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:24.281679 15417 server_base.cc:1061] running on GCE node
I20260430 08:00:24.282357 15417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:24.283263 15417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:24.284428 15417 hybrid_clock.cc:648] HybridClock initialized: now 1777536024284408 us; error 33 us; skew 500 ppm
Apr 30 08:00:24 dist-test-slave-9bxp krb5kdc[15401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536024, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:24.287328 15417 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:24.288486 15417 webserver.cc:492] Webserver started at http://127.13.94.190:46557/ using document root <none> and password file <none>
I20260430 08:00:24.289124 15417 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:24.289175 15417 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:24.289398 15417 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:24.291139 15417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "4cf9e3981063446dae66a6d8efec9caa"
format_stamp: "Formatted at 2026-04-30 08:00:24 on dist-test-slave-9bxp"
server_key: "302faeb435654ef39f8a040edc262e4d"
server_key_iv: "7cd69a247e712889a0e6dc93afde078e"
server_key_version: "encryptionkey@0"
I20260430 08:00:24.291679 15417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "4cf9e3981063446dae66a6d8efec9caa"
format_stamp: "Formatted at 2026-04-30 08:00:24 on dist-test-slave-9bxp"
server_key: "302faeb435654ef39f8a040edc262e4d"
server_key_iv: "7cd69a247e712889a0e6dc93afde078e"
server_key_version: "encryptionkey@0"
I20260430 08:00:24.295289 15417 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260430 08:00:24.297608 15432 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:24.298674 15417 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260430 08:00:24.298820 15417 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "4cf9e3981063446dae66a6d8efec9caa"
format_stamp: "Formatted at 2026-04-30 08:00:24 on dist-test-slave-9bxp"
server_key: "302faeb435654ef39f8a040edc262e4d"
server_key_iv: "7cd69a247e712889a0e6dc93afde078e"
server_key_version: "encryptionkey@0"
I20260430 08:00:24.298935 15417 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:24.314383 15417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:24.317140 15417 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:24.317358 15417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:24.325726 15417 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:46039
I20260430 08:00:24.325735 15494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:46039 every 8 connection(s)
I20260430 08:00:24.326848 15417 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:24.327476 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 15417
I20260430 08:00:24.327610 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:24.328081 13690 external_mini_cluster.cc:1468] Setting key 1a05849e1f4f64d9b5a02e24f60c0467
I20260430 08:00:24.330498 15495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:24.336356 15495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa: Bootstrap starting.
Apr 30 08:00:24 dist-test-slave-9bxp krb5kdc[15401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:24.338570 15495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:24.339263 15495 log.cc:826] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:24.341481 15495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa: No bootstrap required, opened a new log
I20260430 08:00:24.341985 15498 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:24.329444 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.13.94.190:46039)
0430 08:00:24.330299 (+ 855us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:24.330311 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:24.330353 (+ 42us) server_negotiation.cc:408] Connection header received
0430 08:00:24.331224 (+ 871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:24.331250 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:24.331606 (+ 356us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:24.331949 (+ 343us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:24.332882 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.334087 (+ 1205us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:24.334777 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.335042 (+ 265us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:24.337329 (+ 2287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:24.337385 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:24.337396 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:24.337424 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:24.339269 (+ 1845us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.339768 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.339776 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.339783 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.339893 (+ 110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.340259 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.340266 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.340268 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.340569 (+ 301us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:24.340803 (+ 234us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:24.341109 (+ 306us) server_negotiation.cc:300] Negotiation successful
0430 08:00:24.341383 (+ 274us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260430 08:00:24.344497 15495 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf9e3981063446dae66a6d8efec9caa" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46039 } }
I20260430 08:00:24.344676 15495 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:24.344789 15495 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf9e3981063446dae66a6d8efec9caa, State: Initialized, Role: FOLLOWER
I20260430 08:00:24.345158 15495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf9e3981063446dae66a6d8efec9caa" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46039 } }
I20260430 08:00:24.345296 15495 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:24.345355 15495 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:24.345464 15495 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:24.346309 15495 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf9e3981063446dae66a6d8efec9caa" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46039 } }
I20260430 08:00:24.346635 15495 leader_election.cc:304] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cf9e3981063446dae66a6d8efec9caa; no voters:
I20260430 08:00:24.346925 15495 leader_election.cc:290] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:24.347119 15500 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:24.347380 15500 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [term 1 LEADER]: Becoming Leader. State: Replica: 4cf9e3981063446dae66a6d8efec9caa, State: Running, Role: LEADER
I20260430 08:00:24.347692 15500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf9e3981063446dae66a6d8efec9caa" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46039 } }
I20260430 08:00:24.348134 15495 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:24.349426 15502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cf9e3981063446dae66a6d8efec9caa. Latest consensus state: current_term: 1 leader_uuid: "4cf9e3981063446dae66a6d8efec9caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf9e3981063446dae66a6d8efec9caa" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46039 } } }
I20260430 08:00:24.349440 15501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cf9e3981063446dae66a6d8efec9caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cf9e3981063446dae66a6d8efec9caa" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46039 } } }
I20260430 08:00:24.349584 15501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:24.349583 15502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:24.350343 15509 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:24.353665 15509 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:24.359033 15509 catalog_manager.cc:1357] Generated new cluster ID: b75c5c2be3bd4e05aba46d7fb9a05966
I20260430 08:00:24.359129 15509 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:24.387812 15509 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:24.388720 15509 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:24.395229 15509 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa: Generated new TSK 0
I20260430 08:00:24.395819 15509 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:24.456039 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:46039
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:24.560415 15523 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:24.560686 15523 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:24.560817 15523 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:24.564254 15523 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:24.564334 15523 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:24.564392 15523 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260430 08:00:24.564409 15523 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 08:00:24.564456 15523 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:24.568945 15523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.94.190:46039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:24.570050 15523 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:24.570977 15523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:24.577486 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:24.577538 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:24.577643 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:24.577674 15523 server_base.cc:1061] running on GCE node
I20260430 08:00:24.578110 15523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:24.578608 15523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:24.579783 15523 hybrid_clock.cc:648] HybridClock initialized: now 1777536024579767 us; error 38 us; skew 500 ppm
Apr 30 08:00:24 dist-test-slave-9bxp krb5kdc[15401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536024, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:24.582361 15523 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:00:24.583331 15523 webserver.cc:492] Webserver started at http://127.13.94.129:37055/ using document root <none> and password file <none>
I20260430 08:00:24.583863 15523 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:24.583935 15523 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:24.584142 15523 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:24.585858 15523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "96467eab96b84e8c91246a631e9fdc4a"
format_stamp: "Formatted at 2026-04-30 08:00:24 on dist-test-slave-9bxp"
server_key: "86c4c18d85db6e9ac3982fa321686c3e"
server_key_iv: "7bcbfdadf3cfff837dbae02639a77116"
server_key_version: "encryptionkey@0"
I20260430 08:00:24.586344 15523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "96467eab96b84e8c91246a631e9fdc4a"
format_stamp: "Formatted at 2026-04-30 08:00:24 on dist-test-slave-9bxp"
server_key: "86c4c18d85db6e9ac3982fa321686c3e"
server_key_iv: "7bcbfdadf3cfff837dbae02639a77116"
server_key_version: "encryptionkey@0"
I20260430 08:00:24.589466 15523 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260430 08:00:24.591548 15538 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:24.592547 15523 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:24.592734 15523 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "96467eab96b84e8c91246a631e9fdc4a"
format_stamp: "Formatted at 2026-04-30 08:00:24 on dist-test-slave-9bxp"
server_key: "86c4c18d85db6e9ac3982fa321686c3e"
server_key_iv: "7bcbfdadf3cfff837dbae02639a77116"
server_key_version: "encryptionkey@0"
I20260430 08:00:24.592851 15523 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:24.602278 15523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:24.605186 15523 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:24.605409 15523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:24.606848 15523 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:24.606925 15523 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:24.607004 15523 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:24.607061 15523 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 08:00:24 dist-test-slave-9bxp krb5kdc[15401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536024, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:24.618800 15523 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:37421
I20260430 08:00:24.618816 15653 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:37421 every 8 connection(s)
I20260430 08:00:24.619151 15498 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:24.607498 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:60703 (local address 127.13.94.190:46039)
0430 08:00:24.607658 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:24.607665 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:24.608360 (+ 695us) server_negotiation.cc:408] Connection header received
0430 08:00:24.609290 (+ 930us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:24.609296 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:24.609362 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:24.609469 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:24.611120 (+ 1651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.611917 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:24.612825 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.613013 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:24.615330 (+ 2317us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:24.615352 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:24.615355 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:24.615385 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:24.617151 (+ 1766us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.617690 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.617694 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.617696 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.617747 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.618008 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.618011 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.618013 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.618203 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:24.618322 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:24.618858 (+ 536us) server_negotiation.cc:300] Negotiation successful
0430 08:00:24.618977 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 08:00:24.619750 15547 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:24.607784 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46039 (local address 127.13.94.129:60703)
0430 08:00:24.608209 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:24.608246 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:24.609082 (+ 836us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:24.609624 (+ 542us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:24.609632 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:24.610253 (+ 621us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:24.610944 (+ 691us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:24.610967 (+ 23us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.612043 (+ 1076us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:24.612047 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:24.612692 (+ 645us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:24.612703 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.612926 (+ 223us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:24.613604 (+ 678us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:24.613630 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:24.615167 (+ 1537us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:24.617265 (+ 2098us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:24.617272 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:24.617284 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:24.617578 (+ 294us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:24.617835 (+ 257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:24.617837 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:24.617839 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:24.617918 (+ 79us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:24.618355 (+ 437us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:24.618361 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:24.618617 (+ 256us) client_negotiation.cc:770] Sending connection context
0430 08:00:24.618820 (+ 203us) client_negotiation.cc:241] Negotiation successful
0430 08:00:24.619043 (+ 223us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
I20260430 08:00:24.619982 15523 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:00:24.621668 15654 heartbeater.cc:344] Connected to a master server at 127.13.94.190:46039
I20260430 08:00:24.621831 15654 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:24.621835 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 15523
I20260430 08:00:24.621940 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:24.622195 13690 external_mini_cluster.cc:1468] Setting key aceeeba7aff144b0e9b205890b424614
I20260430 08:00:24.622421 15654 heartbeater.cc:507] Master 127.13.94.190:46039 requested a full tablet report, sending...
I20260430 08:00:24.623950 15448 ts_manager.cc:194] Registered new tserver with Master: 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421)
I20260430 08:00:24.625473 15448 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:60703
I20260430 08:00:24.630168 15660 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:24.623020 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46039 (local address 127.13.94.129:52601)
0430 08:00:24.623302 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:24.623319 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:24.623407 (+ 88us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:24.623745 (+ 338us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:24.623749 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:24.623946 (+ 197us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:24.624228 (+ 282us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:24.624256 (+ 28us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.625265 (+ 1009us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:24.625269 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:24.625800 (+ 531us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:24.625809 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.625933 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:24.626492 (+ 559us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:24.626509 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:24.626889 (+ 380us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:24.628893 (+ 2004us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:24.628896 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:24.628898 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:24.629086 (+ 188us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:24.629361 (+ 275us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:24.629363 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:24.629365 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:24.629407 (+ 42us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:24.629776 (+ 369us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:24.629778 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:24.629850 (+ 72us) client_negotiation.cc:770] Sending connection context
0430 08:00:24.629926 (+ 76us) client_negotiation.cc:241] Negotiation successful
0430 08:00:24.630013 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
I20260430 08:00:24.630301 15498 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:24.623240 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:52601 (local address 127.13.94.190:46039)
0430 08:00:24.623396 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:24.623400 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:24.623414 (+ 14us) server_negotiation.cc:408] Connection header received
0430 08:00:24.623498 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:24.623501 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:24.623540 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:24.623641 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:24.624378 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.625139 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:24.625928 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.626171 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:24.627002 (+ 831us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:24.627017 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:24.627022 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:24.627053 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:24.628778 (+ 1725us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.629190 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.629197 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.629201 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.629259 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.629496 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.629502 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.629507 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.629677 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:24.629776 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:24.630008 (+ 232us) server_negotiation.cc:300] Negotiation successful
0430 08:00:24.630122 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:24.671924 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:46039
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:24.777251 15665 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:24.777463 15665 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:24.777530 15665 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:24.780941 15665 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:24.781015 15665 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:24.781071 15665 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260430 08:00:24.781087 15665 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 08:00:24.781129 15665 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:24.785573 15665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.94.190:46039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:24.786643 15665 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:24.787580 15665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:24.794150 15671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:24.794124 15673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:24.794170 15670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:24.794538 15665 server_base.cc:1061] running on GCE node
I20260430 08:00:24.794960 15665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:24.795575 15665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:24.796815 15665 hybrid_clock.cc:648] HybridClock initialized: now 1777536024796780 us; error 49 us; skew 500 ppm
Apr 30 08:00:24 dist-test-slave-9bxp krb5kdc[15401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536024, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:24.799770 15665 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:00:24.801131 15665 webserver.cc:492] Webserver started at http://127.13.94.130:45831/ using document root <none> and password file <none>
I20260430 08:00:24.801710 15665 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:24.801787 15665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:24.802007 15665 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:24.803767 15665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "5bd2f44156784407af1156550cdfacf4"
format_stamp: "Formatted at 2026-04-30 08:00:24 on dist-test-slave-9bxp"
server_key: "61efd027150c29d07f7662ddb999161c"
server_key_iv: "057f2837e0c4e8534abc5bd1f402fd85"
server_key_version: "encryptionkey@0"
I20260430 08:00:24.804266 15665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "5bd2f44156784407af1156550cdfacf4"
format_stamp: "Formatted at 2026-04-30 08:00:24 on dist-test-slave-9bxp"
server_key: "61efd027150c29d07f7662ddb999161c"
server_key_iv: "057f2837e0c4e8534abc5bd1f402fd85"
server_key_version: "encryptionkey@0"
I20260430 08:00:24.807699 15665 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260430 08:00:24.809931 15680 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:24.810946 15665 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:24.811090 15665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "5bd2f44156784407af1156550cdfacf4"
format_stamp: "Formatted at 2026-04-30 08:00:24 on dist-test-slave-9bxp"
server_key: "61efd027150c29d07f7662ddb999161c"
server_key_iv: "057f2837e0c4e8534abc5bd1f402fd85"
server_key_version: "encryptionkey@0"
I20260430 08:00:24.811195 15665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:24.822237 15665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:24.824851 15665 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:24.825053 15665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:24.826550 15665 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:24.826623 15665 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:24.826699 15665 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:24.826751 15665 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 08:00:24 dist-test-slave-9bxp krb5kdc[15401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536024, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:24.838433 15665 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:37527
I20260430 08:00:24.838459 15795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:37527 every 8 connection(s)
I20260430 08:00:24.838821 15498 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:24.827220 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:37905 (local address 127.13.94.190:46039)
0430 08:00:24.827382 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:24.827387 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:24.828100 (+ 713us) server_negotiation.cc:408] Connection header received
0430 08:00:24.829029 (+ 929us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:24.829033 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:24.829101 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:24.829215 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:24.830544 (+ 1329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.831374 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:24.832261 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.832450 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:24.834941 (+ 2491us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:24.834970 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:24.834973 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:24.835006 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:24.836612 (+ 1606us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.837300 (+ 688us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.837303 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.837305 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.837360 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.837689 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.837691 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.837693 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.837831 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:24.837907 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:24.838553 (+ 646us) server_negotiation.cc:300] Negotiation successful
0430 08:00:24.838684 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260430 08:00:24.839483 15689 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:24.827526 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46039 (local address 127.13.94.130:37905)
0430 08:00:24.827948 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:24.827985 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:24.828828 (+ 843us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:24.829434 (+ 606us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:24.829445 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:24.829898 (+ 453us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:24.830391 (+ 493us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:24.830403 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.831500 (+ 1097us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:24.831506 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:24.832148 (+ 642us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:24.832160 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.832372 (+ 212us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:24.833037 (+ 665us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:24.833063 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:24.834785 (+ 1722us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:24.836781 (+ 1996us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:24.836790 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:24.836805 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:24.837177 (+ 372us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:24.837480 (+ 303us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:24.837483 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:24.837485 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:24.837596 (+ 111us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:24.837946 (+ 350us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:24.837954 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:24.838245 (+ 291us) client_negotiation.cc:770] Sending connection context
0430 08:00:24.838493 (+ 248us) client_negotiation.cc:241] Negotiation successful
0430 08:00:24.838725 (+ 232us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260430 08:00:24.839530 15665 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:00:24.840955 15796 heartbeater.cc:344] Connected to a master server at 127.13.94.190:46039
I20260430 08:00:24.841116 15796 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:24.841732 15796 heartbeater.cc:507] Master 127.13.94.190:46039 requested a full tablet report, sending...
I20260430 08:00:24.842674 15448 ts_manager.cc:194] Registered new tserver with Master: 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527)
I20260430 08:00:24.843348 15448 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:37905
I20260430 08:00:24.847764 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 15665
I20260430 08:00:24.847874 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:24.848114 13690 external_mini_cluster.cc:1468] Setting key 4bc5fa0d3f2603fa555c48f793b33c36
I20260430 08:00:24.849047 15498 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:24.843058 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:53981 (local address 127.13.94.190:46039)
0430 08:00:24.843192 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:24.843197 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:24.843373 (+ 176us) server_negotiation.cc:408] Connection header received
0430 08:00:24.843466 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:24.843470 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:24.843515 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:24.843600 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:24.844117 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.844592 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:24.845206 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.845406 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:24.846110 (+ 704us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:24.846132 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:24.846134 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:24.846154 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:24.847255 (+ 1101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.847644 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.847648 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.847650 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.847695 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:24.848028 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:24.848031 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:24.848033 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:24.848211 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:24.848302 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:24.848657 (+ 355us) server_negotiation.cc:300] Negotiation successful
0430 08:00:24.848828 (+ 171us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260430 08:00:24.849123 15802 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:24.842941 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46039 (local address 127.13.94.130:53981)
0430 08:00:24.843265 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:24.843281 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:24.843389 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:24.843615 (+ 226us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:24.843618 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:24.843802 (+ 184us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:24.843992 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:24.843997 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.844716 (+ 719us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:24.844723 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:24.845091 (+ 368us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:24.845097 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:24.845223 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:24.845708 (+ 485us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:24.845720 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:24.845971 (+ 251us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:24.847375 (+ 1404us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:24.847380 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:24.847382 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:24.847546 (+ 164us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:24.847793 (+ 247us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:24.847800 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:24.847801 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:24.847875 (+ 74us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:24.848303 (+ 428us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:24.848308 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:24.848401 (+ 93us) client_negotiation.cc:770] Sending connection context
0430 08:00:24.848529 (+ 128us) client_negotiation.cc:241] Negotiation successful
0430 08:00:24.848653 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:24.897243 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:46039
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:25.012589 15807 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:25.012885 15807 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:25.012996 15807 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:25.016515 15807 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:25.016600 15807 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:25.016685 15807 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260430 08:00:25.016753 15807 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 08:00:25.016842 15807 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:25.021167 15807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.94.190:46039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:25.022325 15807 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:25.023190 15807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:25.029685 15812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:25.029819 15815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:25.029896 15807 server_base.cc:1061] running on GCE node
W20260430 08:00:25.029968 15813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:25.030284 15807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:25.030804 15807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:25.031985 15807 hybrid_clock.cc:648] HybridClock initialized: now 1777536025031968 us; error 32 us; skew 500 ppm
Apr 30 08:00:25 dist-test-slave-9bxp krb5kdc[15401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536025, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:25.034657 15807 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:00:25.035686 15807 webserver.cc:492] Webserver started at http://127.13.94.131:33607/ using document root <none> and password file <none>
I20260430 08:00:25.036232 15807 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:25.036304 15807 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:25.036541 15807 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:25.038328 15807 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "d6741a14d3a1427db23f3d7e537d6def"
format_stamp: "Formatted at 2026-04-30 08:00:25 on dist-test-slave-9bxp"
server_key: "2a339dbff5381d27f5b8ef26e86acc09"
server_key_iv: "e9ae3091ad1d90a90266467a76773ee8"
server_key_version: "encryptionkey@0"
I20260430 08:00:25.038830 15807 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "d6741a14d3a1427db23f3d7e537d6def"
format_stamp: "Formatted at 2026-04-30 08:00:25 on dist-test-slave-9bxp"
server_key: "2a339dbff5381d27f5b8ef26e86acc09"
server_key_iv: "e9ae3091ad1d90a90266467a76773ee8"
server_key_version: "encryptionkey@0"
I20260430 08:00:25.042464 15807 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260430 08:00:25.044816 15822 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:25.045835 15807 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260430 08:00:25.045971 15807 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "d6741a14d3a1427db23f3d7e537d6def"
format_stamp: "Formatted at 2026-04-30 08:00:25 on dist-test-slave-9bxp"
server_key: "2a339dbff5381d27f5b8ef26e86acc09"
server_key_iv: "e9ae3091ad1d90a90266467a76773ee8"
server_key_version: "encryptionkey@0"
I20260430 08:00:25.046082 15807 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:25.059993 15807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:25.063166 15807 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:25.063407 15807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:25.065034 15807 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:25.065114 15807 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:25.065177 15807 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:25.065219 15807 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 08:00:25 dist-test-slave-9bxp krb5kdc[15401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536025, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:25.076683 15807 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:37917
I20260430 08:00:25.076719 15937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:37917 every 8 connection(s)
I20260430 08:00:25.077703 15807 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:00:25.079651 15498 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.065984 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:40771 (local address 127.13.94.190:46039)
0430 08:00:25.066169 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.066174 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.066959 (+ 785us) server_negotiation.cc:408] Connection header received
0430 08:00:25.067961 (+ 1002us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.067965 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.068031 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.068129 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:25.069902 (+ 1773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.070636 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.071498 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.071692 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.075326 (+ 3634us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:25.075356 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:25.075361 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:25.075399 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:25.077405 (+ 2006us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:25.078076 (+ 671us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:25.078087 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:25.078088 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:25.078171 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:25.078669 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:25.078672 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:25.078674 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:25.078833 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:25.078925 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.079430 (+ 505us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.079531 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260430 08:00:25.080332 15831 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.066311 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46039 (local address 127.13.94.131:40771)
0430 08:00:25.066802 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.066839 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.067736 (+ 897us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.068323 (+ 587us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.068335 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.068939 (+ 604us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:25.069667 (+ 728us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.069705 (+ 38us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.070785 (+ 1080us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.070789 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.071357 (+ 568us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.071365 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.071601 (+ 236us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.072368 (+ 767us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:25.072400 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:25.075123 (+ 2723us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:25.077549 (+ 2426us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:25.077556 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:25.077569 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:25.077940 (+ 371us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:25.078303 (+ 363us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:25.078307 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:25.078309 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:25.078433 (+ 124us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:25.078964 (+ 531us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:25.078969 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:25.079183 (+ 214us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.079375 (+ 192us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.079563 (+ 188us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":152,"threads_started":1}
I20260430 08:00:25.081764 15938 heartbeater.cc:344] Connected to a master server at 127.13.94.190:46039
I20260430 08:00:25.081969 15938 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:25.082527 15938 heartbeater.cc:507] Master 127.13.94.190:46039 requested a full tablet report, sending...
I20260430 08:00:25.083459 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 15807
I20260430 08:00:25.083575 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:25.083717 15449 ts_manager.cc:194] Registered new tserver with Master: d6741a14d3a1427db23f3d7e537d6def (127.13.94.131:37917)
I20260430 08:00:25.083854 13690 external_mini_cluster.cc:1468] Setting key 0019b795df12370ddf92c50cc240e623
I20260430 08:00:25.084492 15449 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:40771
I20260430 08:00:25.086081 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 08:00:25.090891 15498 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.083934 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:59817 (local address 127.13.94.190:46039)
0430 08:00:25.084060 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.084065 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.084441 (+ 376us) server_negotiation.cc:408] Connection header received
0430 08:00:25.084588 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.084592 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.084634 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.084750 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:25.085399 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.085874 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.086632 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.086769 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.087757 (+ 988us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:25.087771 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:25.087773 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:25.087796 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:25.089253 (+ 1457us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:25.089720 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:25.089723 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:25.089724 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:25.089766 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:25.090045 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:25.090085 (+ 40us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:25.090086 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:25.090252 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:25.090370 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.090627 (+ 257us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.090732 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260430 08:00:25.090914 15944 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.084063 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46039 (local address 127.13.94.131:59817)
0430 08:00:25.084354 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.084368 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.084484 (+ 116us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.084766 (+ 282us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.084771 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.084977 (+ 206us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:25.085271 (+ 294us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.085280 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.085979 (+ 699us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.085985 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.086495 (+ 510us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.086508 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.086634 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.087175 (+ 541us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:25.087198 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:25.087621 (+ 423us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:25.089368 (+ 1747us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:25.089375 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:25.089380 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:25.089618 (+ 238us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:25.089856 (+ 238us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:25.089863 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:25.089867 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:25.089934 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:25.090362 (+ 428us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:25.090365 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:25.090460 (+ 95us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.090574 (+ 114us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.090724 (+ 150us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
Apr 30 08:00:25 dist-test-slave-9bxp krb5kdc[15401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 08:00:25 dist-test-slave-9bxp krb5kdc[15401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:25.111614 15498 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.103864 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.13.94.190:46039)
0430 08:00:25.104071 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.104074 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.104086 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:00:25.104126 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.104128 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.104171 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.104265 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:25.105178 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.105681 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.106356 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.106565 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.108739 (+ 2174us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:25.108769 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:25.108771 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:25.108800 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:25.110020 (+ 1220us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:25.110475 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:25.110481 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:25.110487 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:25.110536 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:25.110784 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:25.110788 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:25.110790 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:25.110945 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:25.111100 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.111318 (+ 218us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.111432 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260430 08:00:25.114364 15449 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:00:25.116920 15449 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:25.129926 15957 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.125657 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:37917 (local address 127.0.0.1:55906)
0430 08:00:25.126585 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.126598 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.126678 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.127255 (+ 577us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.127258 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.127272 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:25.127481 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.127488 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.128552 (+ 1064us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.128556 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.129524 (+ 968us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.129534 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.129641 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.129656 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.129700 (+ 44us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.129752 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":95,"threads_started":1}
I20260430 08:00:25.130738 15959 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.126000 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.13.94.131:37917)
0430 08:00:25.126295 (+ 295us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.126301 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.126770 (+ 469us) server_negotiation.cc:408] Connection header received
0430 08:00:25.126851 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.126856 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.127002 (+ 146us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.127123 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:25.127602 (+ 479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.128435 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.129671 (+ 1236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.130330 (+ 659us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.130452 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.130519 (+ 67us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.130598 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260430 08:00:25.130736 15956 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.125337 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.13.94.129:37421)
0430 08:00:25.125703 (+ 366us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.125712 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.125738 (+ 26us) server_negotiation.cc:408] Connection header received
0430 08:00:25.126100 (+ 362us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.126106 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.126292 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.126445 (+ 153us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:25.127126 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.128202 (+ 1076us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.129841 (+ 1639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.130375 (+ 534us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.130451 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.130519 (+ 68us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.130603 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":77,"threads_started":1}
I20260430 08:00:25.130982 15955 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.125166 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:37421 (local address 127.0.0.1:58660)
0430 08:00:25.125478 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.125513 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.125887 (+ 374us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.126665 (+ 778us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.126670 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.126688 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:25.126994 (+ 306us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.127001 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.128322 (+ 1321us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.128326 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.129366 (+ 1040us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.129375 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.129481 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.129537 (+ 56us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.129639 (+ 102us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.129713 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260430 08:00:25.131222 15960 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.126284 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.13.94.130:37527)
0430 08:00:25.126973 (+ 689us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.126978 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.126993 (+ 15us) server_negotiation.cc:408] Connection header received
0430 08:00:25.127054 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.127080 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.127289 (+ 209us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.127421 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:25.127766 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.128814 (+ 1048us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.130303 (+ 1489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.130781 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.130917 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.130985 (+ 68us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.131065 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":116,"threads_started":1}
I20260430 08:00:25.131433 15958 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.125851 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:37527 (local address 127.0.0.1:42174)
0430 08:00:25.126702 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.126715 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.126790 (+ 75us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.127424 (+ 634us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.127427 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.127440 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:25.127632 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.127641 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.128940 (+ 1299us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.128943 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.129756 (+ 813us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.129763 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.129850 (+ 87us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.129861 (+ 11us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.129904 (+ 43us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.129952 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":94,"threads_started":1}
I20260430 08:00:25.133785 15730 tablet_service.cc:1511] Processing CreateTablet for tablet bb9c59f7d96444c18f3f7974a4198c15 (DEFAULT_TABLE table=test-table [id=e735ee721f464b76b494c6bb4367fb57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:25.133787 15872 tablet_service.cc:1511] Processing CreateTablet for tablet bb9c59f7d96444c18f3f7974a4198c15 (DEFAULT_TABLE table=test-table [id=e735ee721f464b76b494c6bb4367fb57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:25.133777 15588 tablet_service.cc:1511] Processing CreateTablet for tablet bb9c59f7d96444c18f3f7974a4198c15 (DEFAULT_TABLE table=test-table [id=e735ee721f464b76b494c6bb4367fb57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:25.134598 15730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9c59f7d96444c18f3f7974a4198c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:25.134660 15588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9c59f7d96444c18f3f7974a4198c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:25.134660 15872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb9c59f7d96444c18f3f7974a4198c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:25.139926 15961 tablet_bootstrap.cc:492] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: Bootstrap starting.
I20260430 08:00:25.141007 15962 tablet_bootstrap.cc:492] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: Bootstrap starting.
I20260430 08:00:25.141321 15963 tablet_bootstrap.cc:492] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: Bootstrap starting.
I20260430 08:00:25.142566 15961 tablet_bootstrap.cc:654] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:25.143028 15962 tablet_bootstrap.cc:654] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:25.143150 15963 tablet_bootstrap.cc:654] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:25.143676 15961 log.cc:826] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:25.143886 15962 log.cc:826] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:25.144256 15963 log.cc:826] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:25.145506 15961 tablet_bootstrap.cc:492] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: No bootstrap required, opened a new log
I20260430 08:00:25.145597 15962 tablet_bootstrap.cc:492] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: No bootstrap required, opened a new log
I20260430 08:00:25.145742 15961 ts_tablet_manager.cc:1403] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260430 08:00:25.145779 15962 ts_tablet_manager.cc:1403] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 08:00:25.145857 15963 tablet_bootstrap.cc:492] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: No bootstrap required, opened a new log
I20260430 08:00:25.146019 15963 ts_tablet_manager.cc:1403] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260430 08:00:25.148379 15961 raft_consensus.cc:359] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.148582 15961 raft_consensus.cc:385] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:25.148639 15961 raft_consensus.cc:740] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd2f44156784407af1156550cdfacf4, State: Initialized, Role: FOLLOWER
I20260430 08:00:25.148803 15962 raft_consensus.cc:359] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.148803 15963 raft_consensus.cc:359] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.149001 15962 raft_consensus.cc:385] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:25.149001 15963 raft_consensus.cc:385] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:25.149063 15963 raft_consensus.cc:740] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6741a14d3a1427db23f3d7e537d6def, State: Initialized, Role: FOLLOWER
I20260430 08:00:25.149057 15962 raft_consensus.cc:740] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96467eab96b84e8c91246a631e9fdc4a, State: Initialized, Role: FOLLOWER
I20260430 08:00:25.149099 15961 consensus_queue.cc:260] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.149477 15963 consensus_queue.cc:260] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.149477 15962 consensus_queue.cc:260] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.149927 15796 heartbeater.cc:499] Master 127.13.94.190:46039 was elected leader, sending a full tablet report...
I20260430 08:00:25.150302 15962 ts_tablet_manager.cc:1434] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260430 08:00:25.150302 15963 ts_tablet_manager.cc:1434] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260430 08:00:25.150640 15938 heartbeater.cc:499] Master 127.13.94.190:46039 was elected leader, sending a full tablet report...
I20260430 08:00:25.151046 15654 heartbeater.cc:499] Master 127.13.94.190:46039 was elected leader, sending a full tablet report...
I20260430 08:00:25.151232 15961 ts_tablet_manager.cc:1434] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 08:00:25.230043 15969 raft_consensus.cc:493] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:25.230219 15969 raft_consensus.cc:515] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.231263 15969 leader_election.cc:290] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527)
I20260430 08:00:25.234913 15831 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.231621 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:37421 (local address 127.13.94.131:60909)
0430 08:00:25.231774 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.231791 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.231903 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.232217 (+ 314us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.232221 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.232254 (+ 33us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:25.232490 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.232499 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.233900 (+ 1401us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.233908 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.234504 (+ 596us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.234511 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.234639 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.234658 (+ 19us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.234710 (+ 52us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.234775 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260430 08:00:25.234931 15970 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.231968 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:37527 (local address 127.13.94.131:34585)
0430 08:00:25.232397 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.232408 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.232486 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.232752 (+ 266us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.232756 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.232778 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:25.233137 (+ 359us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.233147 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.233947 (+ 800us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.233950 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.234514 (+ 564us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.234520 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.234652 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.234668 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.234719 (+ 51us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.234769 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":176,"threads_started":1}
I20260430 08:00:25.235363 15956 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.231686 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:60909 (local address 127.13.94.129:37421)
0430 08:00:25.231842 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.231846 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.231881 (+ 35us) server_negotiation.cc:408] Connection header received
0430 08:00:25.232030 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.232033 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.232099 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.232181 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:25.232641 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.233758 (+ 1117us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.234632 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.235125 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.235161 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.235210 (+ 49us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.235265 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260430 08:00:25.235525 15960 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.232077 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:34585 (local address 127.13.94.130:37527)
0430 08:00:25.232226 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.232230 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.232490 (+ 260us) server_negotiation.cc:408] Connection header received
0430 08:00:25.232555 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.232561 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.232622 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.232690 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:25.233300 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.233793 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.234660 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.235195 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.235245 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.235305 (+ 60us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.235368 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260430 08:00:25.236069 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9c59f7d96444c18f3f7974a4198c15" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a" is_pre_election: true
I20260430 08:00:25.236167 15750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9c59f7d96444c18f3f7974a4198c15" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd2f44156784407af1156550cdfacf4" is_pre_election: true
I20260430 08:00:25.236397 15608 raft_consensus.cc:2468] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 0.
I20260430 08:00:25.236457 15750 raft_consensus.cc:2468] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 0.
I20260430 08:00:25.236927 15824 leader_election.cc:304] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96467eab96b84e8c91246a631e9fdc4a, d6741a14d3a1427db23f3d7e537d6def; no voters:
I20260430 08:00:25.237195 15969 raft_consensus.cc:2804] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:25.237262 15969 raft_consensus.cc:493] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:25.237291 15969 raft_consensus.cc:3060] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:25.238263 15969 raft_consensus.cc:515] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.238627 15969 leader_election.cc:290] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 election: Requested vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527)
I20260430 08:00:25.239116 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9c59f7d96444c18f3f7974a4198c15" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a"
I20260430 08:00:25.239166 15750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb9c59f7d96444c18f3f7974a4198c15" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd2f44156784407af1156550cdfacf4"
I20260430 08:00:25.239253 15608 raft_consensus.cc:3060] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:25.239292 15750 raft_consensus.cc:3060] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:25.240468 15608 raft_consensus.cc:2468] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 1.
I20260430 08:00:25.240546 15750 raft_consensus.cc:2468] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 1.
I20260430 08:00:25.240797 15824 leader_election.cc:304] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96467eab96b84e8c91246a631e9fdc4a, d6741a14d3a1427db23f3d7e537d6def; no voters:
I20260430 08:00:25.240960 15969 raft_consensus.cc:2804] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:25.241181 15969 raft_consensus.cc:697] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 1 LEADER]: Becoming Leader. State: Replica: d6741a14d3a1427db23f3d7e537d6def, State: Running, Role: LEADER
I20260430 08:00:25.241510 15969 consensus_queue.cc:237] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.244517 15447 catalog_manager.cc:5671] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def reported cstate change: term changed from 0 to 1, leader changed from <none> to d6741a14d3a1427db23f3d7e537d6def (127.13.94.131). New cstate: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } health_report { overall_health: HEALTHY } } }
Apr 30 08:00:25 dist-test-slave-9bxp krb5kdc[15401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536024, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:25.314505 15978 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.306622 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46039 (local address 127.0.0.1:53818)
0430 08:00:25.306924 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.306938 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.307052 (+ 114us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.307326 (+ 274us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.307330 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.307759 (+ 429us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:25.307955 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.307961 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.308829 (+ 868us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.308831 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.309233 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.309239 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.309338 (+ 99us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.309935 (+ 597us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:25.309949 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:25.311488 (+ 1539us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:25.313167 (+ 1679us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:25.313174 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:25.313178 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:25.313404 (+ 226us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:25.313634 (+ 230us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:25.313637 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:25.313639 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:25.313689 (+ 50us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:25.314011 (+ 322us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:25.314016 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:25.314114 (+ 98us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.314203 (+ 89us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.314324 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":143,"threads_started":1}
I20260430 08:00:25.314507 15498 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.306651 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.13.94.190:46039)
0430 08:00:25.306814 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.306819 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.307060 (+ 241us) server_negotiation.cc:408] Connection header received
0430 08:00:25.307152 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.307155 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.307209 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.307288 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:25.308123 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.308638 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.309346 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.309536 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.311611 (+ 2075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:25.311633 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:25.311636 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:25.311659 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:25.313039 (+ 1380us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:25.313492 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:25.313494 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:25.313495 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:25.313533 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:25.313771 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:25.313773 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:25.313775 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:25.313909 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:25.313994 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.314259 (+ 265us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.314382 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":58}
I20260430 08:00:25.318787 15447 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:53818:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260430 08:00:25.319561 15447 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:25.324958 15588 tablet_service.cc:1511] Processing CreateTablet for tablet 4ee2d306be804d359143ce337f90f4b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4813e027ce94e019f2acf66aaaf2fbd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 08:00:25.325126 15730 tablet_service.cc:1511] Processing CreateTablet for tablet 4ee2d306be804d359143ce337f90f4b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4813e027ce94e019f2acf66aaaf2fbd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 08:00:25.325717 15588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee2d306be804d359143ce337f90f4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:25.325717 15730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee2d306be804d359143ce337f90f4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:25.325460 15872 tablet_service.cc:1511] Processing CreateTablet for tablet 4ee2d306be804d359143ce337f90f4b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4813e027ce94e019f2acf66aaaf2fbd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 08:00:25.325961 15872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee2d306be804d359143ce337f90f4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:25.328450 15962 tablet_bootstrap.cc:492] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: Bootstrap starting.
W20260430 08:00:25.328608 15939 tablet.cc:2404] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:00:25.329510 15962 tablet_bootstrap.cc:654] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:25.330260 15963 tablet_bootstrap.cc:492] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: Bootstrap starting.
I20260430 08:00:25.331130 15963 tablet_bootstrap.cc:654] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:25.332064 15962 tablet_bootstrap.cc:492] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: No bootstrap required, opened a new log
I20260430 08:00:25.332201 15962 ts_tablet_manager.cc:1403] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260430 08:00:25.332547 15963 tablet_bootstrap.cc:492] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: No bootstrap required, opened a new log
I20260430 08:00:25.332633 15963 ts_tablet_manager.cc:1403] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:25.332760 15962 raft_consensus.cc:359] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.332885 15962 raft_consensus.cc:385] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:25.332926 15962 raft_consensus.cc:740] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96467eab96b84e8c91246a631e9fdc4a, State: Initialized, Role: FOLLOWER
I20260430 08:00:25.333060 15962 consensus_queue.cc:260] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.333153 15963 raft_consensus.cc:359] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.333241 15963 raft_consensus.cc:385] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:25.333278 15963 raft_consensus.cc:740] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6741a14d3a1427db23f3d7e537d6def, State: Initialized, Role: FOLLOWER
I20260430 08:00:25.333417 15963 consensus_queue.cc:260] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.333600 15968 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.333719 15968 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:25.333712 15963 ts_tablet_manager.cc:1434] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:25.333945 15962 ts_tablet_manager.cc:1434] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:25.334098 15969 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.334216 15969 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:25.334729 15961 tablet_bootstrap.cc:492] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: Bootstrap starting.
I20260430 08:00:25.335873 15961 tablet_bootstrap.cc:654] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:25.337317 15961 tablet_bootstrap.cc:492] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: No bootstrap required, opened a new log
I20260430 08:00:25.337424 15961 ts_tablet_manager.cc:1403] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 08:00:25.337843 15961 raft_consensus.cc:359] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.337929 15961 raft_consensus.cc:385] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:25.337988 15961 raft_consensus.cc:740] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd2f44156784407af1156550cdfacf4, State: Initialized, Role: FOLLOWER
I20260430 08:00:25.338142 15961 consensus_queue.cc:260] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.338881 15961 ts_tablet_manager.cc:1434] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:25.338423 15967 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.338994 15967 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: This TxnStatusTablet replica's current role is: FOLLOWER
W20260430 08:00:25.340590 15797 tablet.cc:2404] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:00:25.371321 15655 tablet.cc:2404] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:00:25.393059 15969 raft_consensus.cc:493] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:25.393217 15969 raft_consensus.cc:515] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.393700 15969 leader_election.cc:290] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527)
I20260430 08:00:25.394209 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee2d306be804d359143ce337f90f4b7" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a" is_pre_election: true
I20260430 08:00:25.394209 15750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee2d306be804d359143ce337f90f4b7" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd2f44156784407af1156550cdfacf4" is_pre_election: true
I20260430 08:00:25.394359 15608 raft_consensus.cc:2468] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 0.
I20260430 08:00:25.394359 15750 raft_consensus.cc:2468] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 0.
I20260430 08:00:25.394625 15824 leader_election.cc:304] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd2f44156784407af1156550cdfacf4, d6741a14d3a1427db23f3d7e537d6def; no voters:
I20260430 08:00:25.394795 15969 raft_consensus.cc:2804] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:25.394852 15969 raft_consensus.cc:493] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:25.394889 15969 raft_consensus.cc:3060] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:25.395764 15969 raft_consensus.cc:515] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.396082 15969 leader_election.cc:290] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 election: Requested vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527)
I20260430 08:00:25.396472 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee2d306be804d359143ce337f90f4b7" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a"
I20260430 08:00:25.396471 15750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ee2d306be804d359143ce337f90f4b7" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd2f44156784407af1156550cdfacf4"
I20260430 08:00:25.396586 15750 raft_consensus.cc:3060] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:25.396586 15608 raft_consensus.cc:3060] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:25.397660 15750 raft_consensus.cc:2468] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 1.
I20260430 08:00:25.397660 15608 raft_consensus.cc:2468] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 1.
I20260430 08:00:25.398046 15824 leader_election.cc:304] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd2f44156784407af1156550cdfacf4, d6741a14d3a1427db23f3d7e537d6def; no voters:
I20260430 08:00:25.398259 15969 raft_consensus.cc:2804] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:25.398365 15969 raft_consensus.cc:697] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [term 1 LEADER]: Becoming Leader. State: Replica: d6741a14d3a1427db23f3d7e537d6def, State: Running, Role: LEADER
I20260430 08:00:25.398528 15969 consensus_queue.cc:237] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.399260 15971 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: TxnStatusTablet state changed. Reason: New leader d6741a14d3a1427db23f3d7e537d6def. Latest consensus state: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.399359 15971 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: This TxnStatusTablet replica's current role is: LEADER
I20260430 08:00:25.399685 15983 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260430 08:00:25.399911 15983 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260430 08:00:25.400466 15447 catalog_manager.cc:5671] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def reported cstate change: term changed from 0 to 1, leader changed from <none> to d6741a14d3a1427db23f3d7e537d6def (127.13.94.131). New cstate: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } health_report { overall_health: HEALTHY } } }
I20260430 08:00:25.412482 15959 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.409160 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.13.94.131:37917)
0430 08:00:25.409310 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.409313 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.409392 (+ 79us) server_negotiation.cc:408] Connection header received
0430 08:00:25.409567 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.409570 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.409620 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.409698 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:25.410116 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.410648 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.411590 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.411732 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.411774 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:25.412109 (+ 335us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:25.412189 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.412344 (+ 155us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.412383 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260430 08:00:25.412508 15978 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.409071 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:37917 (local address 127.0.0.1:55918)
0430 08:00:25.409315 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.409332 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.409452 (+ 120us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.409731 (+ 279us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.409734 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.409743 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:00:25.410003 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.410011 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.410762 (+ 751us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.410767 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.411460 (+ 693us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.411473 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.411608 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.411654 (+ 46us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:00:25.412216 (+ 562us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:00:25.412228 (+ 12us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.412307 (+ 79us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.412368 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260430 08:00:25.417732 15750 raft_consensus.cc:1275] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Refusing update from remote peer d6741a14d3a1427db23f3d7e537d6def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:25.417732 15608 raft_consensus.cc:1275] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Refusing update from remote peer d6741a14d3a1427db23f3d7e537d6def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:25.418397 15969 consensus_queue.cc:1048] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:25.418666 15971 consensus_queue.cc:1048] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def [LEADER]: Connected to new peer: Peer: permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:25.422793 15968 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: TxnStatusTablet state changed. Reason: New leader d6741a14d3a1427db23f3d7e537d6def. Latest consensus state: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.422919 15968 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:25.423287 15967 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: TxnStatusTablet state changed. Reason: New leader d6741a14d3a1427db23f3d7e537d6def. Latest consensus state: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.423391 15967 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:25.427088 15991 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.427217 15991 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: This TxnStatusTablet replica's current role is: LEADER
I20260430 08:00:25.427238 15969 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.427335 15969 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P d6741a14d3a1427db23f3d7e537d6def: This TxnStatusTablet replica's current role is: LEADER
I20260430 08:00:25.427430 15968 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.427481 15986 mvcc.cc:204] Tried to move back new op lower bound from 7280787560103612416 to 7280787560033873920. Current Snapshot: MvccSnapshot[applied={T|T < 7280787560103612416}]
I20260430 08:00:25.427524 15968 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P 96467eab96b84e8c91246a631e9fdc4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:25.431839 15967 tablet_replica.cc:442] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } } }
I20260430 08:00:25.431969 15967 tablet_replica.cc:445] T 4ee2d306be804d359143ce337f90f4b7 P 5bd2f44156784407af1156550cdfacf4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:00:25.437570 15959 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.434480 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.13.94.131:37917)
0430 08:00:25.434597 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.434601 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.434686 (+ 85us) server_negotiation.cc:408] Connection header received
0430 08:00:25.434834 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.434837 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.434893 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.434977 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:25.435400 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.436188 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.436877 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.437079 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.437122 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:25.437202 (+ 80us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:25.437282 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.437395 (+ 113us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.437439 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260430 08:00:25.448478 15959 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.445276 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:56387 (local address 127.13.94.131:37917)
0430 08:00:25.445414 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:25.445418 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:25.445847 (+ 429us) server_negotiation.cc:408] Connection header received
0430 08:00:25.445952 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:25.445956 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:25.446016 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:25.446098 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:25.446408 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.447128 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.447729 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.447909 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.447948 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:25.448034 (+ 86us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:25.448112 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:25.448292 (+ 180us) server_negotiation.cc:300] Negotiation successful
0430 08:00:25.448330 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260430 08:00:25.448478 15944 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:25.445621 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:37917 (local address 127.13.94.131:56387)
0430 08:00:25.445762 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:25.445778 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:25.445866 (+ 88us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:25.446127 (+ 261us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:25.446129 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:25.446136 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:00:25.446311 (+ 175us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.446316 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.447234 (+ 918us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:25.447236 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:25.447633 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:25.447638 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:25.447721 (+ 83us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:25.447752 (+ 31us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:00:25.448183 (+ 431us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:00:25.448189 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 08:00:25.448261 (+ 72us) client_negotiation.cc:241] Negotiation successful
0430 08:00:25.448312 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260430 08:00:25.456182 15608 raft_consensus.cc:1275] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Refusing update from remote peer d6741a14d3a1427db23f3d7e537d6def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:25.456167 15750 raft_consensus.cc:1275] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Refusing update from remote peer d6741a14d3a1427db23f3d7e537d6def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:25.456641 15991 consensus_queue.cc:1048] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:25.456794 15969 consensus_queue.cc:1048] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [LEADER]: Connected to new peer: Peer: permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:25.462430 15988 mvcc.cc:204] Tried to move back new op lower bound from 7280787560263553024 to 7280787559392034816. Current Snapshot: MvccSnapshot[applied={T|T < 7280787560263553024}]
I20260430 08:00:25.463141 15987 mvcc.cc:204] Tried to move back new op lower bound from 7280787560263553024 to 7280787559392034816. Current Snapshot: MvccSnapshot[applied={T|T < 7280787560263553024}]
I20260430 08:00:25.501238 15448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:25.501422 15448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:25.504273 15448 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa: Sending DeleteTablet for 3 replicas of tablet bb9c59f7d96444c18f3f7974a4198c15
I20260430 08:00:25.505170 15588 tablet_service.cc:1558] Processing DeleteTablet for tablet bb9c59f7d96444c18f3f7974a4198c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:25 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:58660
I20260430 08:00:25.505280 15730 tablet_service.cc:1558] Processing DeleteTablet for tablet bb9c59f7d96444c18f3f7974a4198c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:25 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42174
I20260430 08:00:25.505867 15871 tablet_service.cc:1558] Processing DeleteTablet for tablet bb9c59f7d96444c18f3f7974a4198c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:25 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:55906
I20260430 08:00:25.505857 15448 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 08:00:25.505923 16003 tablet_replica.cc:333] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: stopping tablet replica
W20260430 08:00:25.506309 15448 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:25.506338 16004 tablet_replica.cc:333] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: stopping tablet replica
I20260430 08:00:25.506484 16003 raft_consensus.cc:2243] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:25.506572 16004 raft_consensus.cc:2243] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:00:25.506781 16003 raft_consensus.cc:2272] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:25.507304 16002 tablet_replica.cc:333] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: stopping tablet replica
I20260430 08:00:25.507530 16002 raft_consensus.cc:2243] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:25.506961 16004 raft_consensus.cc:2272] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:25.511657 16003 ts_tablet_manager.cc:1916] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:25.512264 16002 raft_consensus.cc:2272] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:25.513664 15588 tablet_service.cc:1511] Processing CreateTablet for tablet 9edc12deb29b49d6b0ff8561e86567ef (DEFAULT_TABLE table=test-table [id=9c5eda24da164e018f7ece4875658042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:25.513967 15588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9edc12deb29b49d6b0ff8561e86567ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:25.515118 16003 ts_tablet_manager.cc:1929] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:00:25.515349 16003 log.cc:1199] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/bb9c59f7d96444c18f3f7974a4198c15
I20260430 08:00:25.515684 15871 tablet_service.cc:1511] Processing CreateTablet for tablet 9edc12deb29b49d6b0ff8561e86567ef (DEFAULT_TABLE table=test-table [id=9c5eda24da164e018f7ece4875658042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:25.515874 16003 ts_tablet_manager.cc:1950] T bb9c59f7d96444c18f3f7974a4198c15 P 5bd2f44156784407af1156550cdfacf4: Deleting consensus metadata
I20260430 08:00:25.515987 15871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9edc12deb29b49d6b0ff8561e86567ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:25.516371 15962 tablet_bootstrap.cc:492] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: Bootstrap starting.
I20260430 08:00:25.517679 15962 tablet_bootstrap.cc:654] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:25.518044 15963 tablet_bootstrap.cc:492] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: Bootstrap starting.
I20260430 08:00:25.518740 15730 tablet_service.cc:1511] Processing CreateTablet for tablet 9edc12deb29b49d6b0ff8561e86567ef (DEFAULT_TABLE table=test-table [id=9c5eda24da164e018f7ece4875658042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:25.518970 15730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9edc12deb29b49d6b0ff8561e86567ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:25.519062 15963 tablet_bootstrap.cc:654] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:25.520280 16002 ts_tablet_manager.cc:1916] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:25.521021 15961 tablet_bootstrap.cc:492] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: Bootstrap starting.
I20260430 08:00:25.521582 15963 tablet_bootstrap.cc:492] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: No bootstrap required, opened a new log
I20260430 08:00:25.521667 15963 ts_tablet_manager.cc:1403] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260430 08:00:25.521950 15961 tablet_bootstrap.cc:654] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:25.522186 15963 raft_consensus.cc:359] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.522328 15963 raft_consensus.cc:385] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:25.522365 15963 raft_consensus.cc:740] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6741a14d3a1427db23f3d7e537d6def, State: Initialized, Role: FOLLOWER
I20260430 08:00:25.522490 15963 consensus_queue.cc:260] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.522737 15963 ts_tablet_manager.cc:1434] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:25.522831 15434 catalog_manager.cc:5002] TS 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527): tablet bb9c59f7d96444c18f3f7974a4198c15 (table test-table [id=e735ee721f464b76b494c6bb4367fb57]) successfully deleted
I20260430 08:00:25.523330 16004 ts_tablet_manager.cc:1916] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:25.523965 16002 ts_tablet_manager.cc:1929] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:00:25.524078 16002 log.cc:1199] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/bb9c59f7d96444c18f3f7974a4198c15
I20260430 08:00:25.524441 16002 ts_tablet_manager.cc:1950] T bb9c59f7d96444c18f3f7974a4198c15 P 96467eab96b84e8c91246a631e9fdc4a: Deleting consensus metadata
I20260430 08:00:25.525182 15962 tablet_bootstrap.cc:492] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: No bootstrap required, opened a new log
I20260430 08:00:25.525285 15962 ts_tablet_manager.cc:1403] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260430 08:00:25.525804 15962 raft_consensus.cc:359] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.525928 15962 raft_consensus.cc:385] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:25.525995 15962 raft_consensus.cc:740] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96467eab96b84e8c91246a631e9fdc4a, State: Initialized, Role: FOLLOWER
I20260430 08:00:25.526002 16004 ts_tablet_manager.cc:1929] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:00:25.526101 16004 log.cc:1199] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/bb9c59f7d96444c18f3f7974a4198c15
I20260430 08:00:25.526165 15962 consensus_queue.cc:260] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.526194 15434 catalog_manager.cc:5002] TS 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421): tablet bb9c59f7d96444c18f3f7974a4198c15 (table test-table [id=e735ee721f464b76b494c6bb4367fb57]) successfully deleted
I20260430 08:00:25.526398 16004 ts_tablet_manager.cc:1950] T bb9c59f7d96444c18f3f7974a4198c15 P d6741a14d3a1427db23f3d7e537d6def: Deleting consensus metadata
I20260430 08:00:25.526896 15962 ts_tablet_manager.cc:1434] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:25.527391 15435 catalog_manager.cc:5002] TS d6741a14d3a1427db23f3d7e537d6def (127.13.94.131:37917): tablet bb9c59f7d96444c18f3f7974a4198c15 (table test-table [id=e735ee721f464b76b494c6bb4367fb57]) successfully deleted
I20260430 08:00:25.528527 15961 tablet_bootstrap.cc:492] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: No bootstrap required, opened a new log
I20260430 08:00:25.528632 15961 ts_tablet_manager.cc:1403] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260430 08:00:25.529209 15961 raft_consensus.cc:359] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.529340 15961 raft_consensus.cc:385] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:25.529382 15961 raft_consensus.cc:740] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd2f44156784407af1156550cdfacf4, State: Initialized, Role: FOLLOWER
I20260430 08:00:25.529539 15961 consensus_queue.cc:260] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.529839 15961 ts_tablet_manager.cc:1434] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:25.880270 16001 raft_consensus.cc:493] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:25.880416 16001 raft_consensus.cc:515] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.881035 16001 leader_election.cc:290] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527)
I20260430 08:00:25.881542 15750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9edc12deb29b49d6b0ff8561e86567ef" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd2f44156784407af1156550cdfacf4" is_pre_election: true
I20260430 08:00:25.881526 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9edc12deb29b49d6b0ff8561e86567ef" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a" is_pre_election: true
I20260430 08:00:25.881690 15750 raft_consensus.cc:2468] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 0.
I20260430 08:00:25.881690 15608 raft_consensus.cc:2468] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 0.
I20260430 08:00:25.882040 15824 leader_election.cc:304] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96467eab96b84e8c91246a631e9fdc4a, d6741a14d3a1427db23f3d7e537d6def; no voters:
I20260430 08:00:25.882225 16001 raft_consensus.cc:2804] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:25.882285 16001 raft_consensus.cc:493] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:25.882325 16001 raft_consensus.cc:3060] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:25.883424 16001 raft_consensus.cc:515] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.883773 16001 leader_election.cc:290] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 election: Requested vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527)
I20260430 08:00:25.884181 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9edc12deb29b49d6b0ff8561e86567ef" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a"
I20260430 08:00:25.884217 15750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9edc12deb29b49d6b0ff8561e86567ef" candidate_uuid: "d6741a14d3a1427db23f3d7e537d6def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd2f44156784407af1156550cdfacf4"
I20260430 08:00:25.884320 15608 raft_consensus.cc:3060] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:25.884335 15750 raft_consensus.cc:3060] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:25.885490 15750 raft_consensus.cc:2468] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 1.
I20260430 08:00:25.885493 15608 raft_consensus.cc:2468] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6741a14d3a1427db23f3d7e537d6def in term 1.
I20260430 08:00:25.885825 15824 leader_election.cc:304] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96467eab96b84e8c91246a631e9fdc4a, d6741a14d3a1427db23f3d7e537d6def; no voters:
I20260430 08:00:25.886016 16001 raft_consensus.cc:2804] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:25.886121 16001 raft_consensus.cc:697] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 1 LEADER]: Becoming Leader. State: Replica: d6741a14d3a1427db23f3d7e537d6def, State: Running, Role: LEADER
I20260430 08:00:25.886299 16001 consensus_queue.cc:237] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:25.887910 15447 catalog_manager.cc:5671] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def reported cstate change: term changed from 0 to 1, leader changed from <none> to d6741a14d3a1427db23f3d7e537d6def (127.13.94.131). New cstate: current_term: 1 leader_uuid: "d6741a14d3a1427db23f3d7e537d6def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } health_report { overall_health: HEALTHY } } }
I20260430 08:00:25.989272 15750 raft_consensus.cc:1275] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Refusing update from remote peer d6741a14d3a1427db23f3d7e537d6def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:25.989360 15608 raft_consensus.cc:1275] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Refusing update from remote peer d6741a14d3a1427db23f3d7e537d6def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:25.989755 15971 consensus_queue.cc:1048] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [LEADER]: Connected to new peer: Peer: permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:25.989917 15969 consensus_queue.cc:1048] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260430 08:00:26.005715 15986 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260430 08:00:27.557305 15448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:27.557467 15448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:27.559768 15448 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa: Sending DeleteTablet for 3 replicas of tablet 9edc12deb29b49d6b0ff8561e86567ef
I20260430 08:00:27.560376 15730 tablet_service.cc:1558] Processing DeleteTablet for tablet 9edc12deb29b49d6b0ff8561e86567ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:27 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42174
I20260430 08:00:27.560400 15588 tablet_service.cc:1558] Processing DeleteTablet for tablet 9edc12deb29b49d6b0ff8561e86567ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:27 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:58660
I20260430 08:00:27.560516 15872 tablet_service.cc:1558] Processing DeleteTablet for tablet 9edc12deb29b49d6b0ff8561e86567ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:27 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:55906
I20260430 08:00:27.560985 16035 tablet_replica.cc:333] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: stopping tablet replica
I20260430 08:00:27.561167 16035 raft_consensus.cc:2243] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:00:27.561386 16035 raft_consensus.cc:2272] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:27.562016 16035 ts_tablet_manager.cc:1916] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:27.563521 16033 tablet_replica.cc:333] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: stopping tablet replica
I20260430 08:00:27.563669 16033 raft_consensus.cc:2243] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:27.563802 16033 raft_consensus.cc:2272] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:27.564061 16034 tablet_replica.cc:333] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: stopping tablet replica
I20260430 08:00:27.564237 16034 raft_consensus.cc:2243] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:27.564374 16034 raft_consensus.cc:2272] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:27.564523 16033 ts_tablet_manager.cc:1916] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:27.564906 16035 ts_tablet_manager.cc:1929] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260430 08:00:27.564991 16035 log.cc:1199] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/9edc12deb29b49d6b0ff8561e86567ef
I20260430 08:00:27.565042 16034 ts_tablet_manager.cc:1916] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:27.565302 16035 ts_tablet_manager.cc:1950] T 9edc12deb29b49d6b0ff8561e86567ef P d6741a14d3a1427db23f3d7e537d6def: Deleting consensus metadata
I20260430 08:00:27.566064 15435 catalog_manager.cc:5002] TS d6741a14d3a1427db23f3d7e537d6def (127.13.94.131:37917): tablet 9edc12deb29b49d6b0ff8561e86567ef (table test-table [id=9c5eda24da164e018f7ece4875658042]) successfully deleted
I20260430 08:00:27.566790 16033 ts_tablet_manager.cc:1929] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260430 08:00:27.566881 16033 log.cc:1199] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/9edc12deb29b49d6b0ff8561e86567ef
I20260430 08:00:27.567162 16033 ts_tablet_manager.cc:1950] T 9edc12deb29b49d6b0ff8561e86567ef P 5bd2f44156784407af1156550cdfacf4: Deleting consensus metadata
I20260430 08:00:27.567170 16034 ts_tablet_manager.cc:1929] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260430 08:00:27.567226 16034 log.cc:1199] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/9edc12deb29b49d6b0ff8561e86567ef
I20260430 08:00:27.567519 16034 ts_tablet_manager.cc:1950] T 9edc12deb29b49d6b0ff8561e86567ef P 96467eab96b84e8c91246a631e9fdc4a: Deleting consensus metadata
I20260430 08:00:27.568200 15434 catalog_manager.cc:5002] TS 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421): tablet 9edc12deb29b49d6b0ff8561e86567ef (table test-table [id=9c5eda24da164e018f7ece4875658042]) successfully deleted
I20260430 08:00:27.568403 15434 catalog_manager.cc:5002] TS 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527): tablet 9edc12deb29b49d6b0ff8561e86567ef (table test-table [id=9c5eda24da164e018f7ece4875658042]) successfully deleted
Apr 30 08:00:27 dist-test-slave-9bxp krb5kdc[15401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536027, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 30 08:00:27 dist-test-slave-9bxp krb5kdc[15401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536027, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:27.595685 16043 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:27.583216 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.13.94.190:46039)
0430 08:00:27.583542 (+ 326us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:27.583546 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:27.583563 (+ 17us) server_negotiation.cc:408] Connection header received
0430 08:00:27.583691 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:27.583694 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:27.583740 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:27.583845 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:27.584644 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.585262 (+ 618us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:27.585915 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.586132 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:27.592727 (+ 6595us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:27.592758 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:27.592761 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:27.592789 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:27.594085 (+ 1296us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:27.594511 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:27.594515 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:27.594517 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:27.594569 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:27.594810 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:27.594813 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:27.594815 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:27.594967 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:27.595111 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:27.595354 (+ 243us) server_negotiation.cc:300] Negotiation successful
0430 08:00:27.595504 (+ 150us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260430 08:00:27.598243 15448 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:00:27.598662 15448 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:27.603677 15588 tablet_service.cc:1511] Processing CreateTablet for tablet 4e15566f01b542bdb88d63f25d6e75b3 (DEFAULT_TABLE table=test-table [id=8db4943256e94d8ea558edf5081bb790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:27.604012 15588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e15566f01b542bdb88d63f25d6e75b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:27.604087 15730 tablet_service.cc:1511] Processing CreateTablet for tablet 4e15566f01b542bdb88d63f25d6e75b3 (DEFAULT_TABLE table=test-table [id=8db4943256e94d8ea558edf5081bb790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:27.604066 15872 tablet_service.cc:1511] Processing CreateTablet for tablet 4e15566f01b542bdb88d63f25d6e75b3 (DEFAULT_TABLE table=test-table [id=8db4943256e94d8ea558edf5081bb790]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:27.604358 15872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e15566f01b542bdb88d63f25d6e75b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:27.604355 15730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e15566f01b542bdb88d63f25d6e75b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:27.606251 16044 tablet_bootstrap.cc:492] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: Bootstrap starting.
I20260430 08:00:27.606817 16046 tablet_bootstrap.cc:492] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: Bootstrap starting.
I20260430 08:00:27.606871 16045 tablet_bootstrap.cc:492] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: Bootstrap starting.
I20260430 08:00:27.607232 16044 tablet_bootstrap.cc:654] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:27.607743 16045 tablet_bootstrap.cc:654] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:27.607743 16046 tablet_bootstrap.cc:654] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:27.608462 16044 tablet_bootstrap.cc:492] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: No bootstrap required, opened a new log
I20260430 08:00:27.608562 16044 ts_tablet_manager.cc:1403] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:27.609030 16046 tablet_bootstrap.cc:492] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: No bootstrap required, opened a new log
I20260430 08:00:27.609161 16046 ts_tablet_manager.cc:1403] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:27.609408 16045 tablet_bootstrap.cc:492] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: No bootstrap required, opened a new log
I20260430 08:00:27.609499 16045 ts_tablet_manager.cc:1403] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260430 08:00:27.609514 16044 raft_consensus.cc:359] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:27.609673 16044 raft_consensus.cc:385] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:27.609741 16044 raft_consensus.cc:740] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6741a14d3a1427db23f3d7e537d6def, State: Initialized, Role: FOLLOWER
I20260430 08:00:27.609748 16046 raft_consensus.cc:359] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:27.609933 16046 raft_consensus.cc:385] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:27.609915 16044 consensus_queue.cc:260] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:27.609997 16046 raft_consensus.cc:740] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd2f44156784407af1156550cdfacf4, State: Initialized, Role: FOLLOWER
I20260430 08:00:27.609972 16045 raft_consensus.cc:359] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:27.610098 16045 raft_consensus.cc:385] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:27.610193 16045 raft_consensus.cc:740] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96467eab96b84e8c91246a631e9fdc4a, State: Initialized, Role: FOLLOWER
I20260430 08:00:27.610154 16046 consensus_queue.cc:260] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:27.610283 16044 ts_tablet_manager.cc:1434] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:27.610431 16045 consensus_queue.cc:260] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:27.610545 16046 ts_tablet_manager.cc:1434] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:27.611178 16045 ts_tablet_manager.cc:1434] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:27.939764 16050 raft_consensus.cc:493] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:27.939967 16050 raft_consensus.cc:515] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:27.941190 16050 leader_election.cc:290] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), d6741a14d3a1427db23f3d7e537d6def (127.13.94.131:37917)
I20260430 08:00:27.945140 16055 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:27.941982 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:37917 (local address 127.13.94.130:36749)
0430 08:00:27.942391 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:27.942406 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:27.942503 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:27.942840 (+ 337us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:27.942844 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:27.942903 (+ 59us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:27.943213 (+ 310us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:27.943222 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.943967 (+ 745us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:27.943972 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:27.944620 (+ 648us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:27.944629 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.944809 (+ 180us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:27.944829 (+ 20us) client_negotiation.cc:770] Sending connection context
0430 08:00:27.944892 (+ 63us) client_negotiation.cc:241] Negotiation successful
0430 08:00:27.944950 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":106,"threads_started":1}
I20260430 08:00:27.945188 16052 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:27.941388 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:37421 (local address 127.13.94.130:43923)
0430 08:00:27.941768 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:27.941785 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:27.941898 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:27.942414 (+ 516us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:27.942421 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:27.942450 (+ 29us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:27.942815 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:27.942831 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.944086 (+ 1255us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:27.944089 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:27.944659 (+ 570us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:27.944666 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.944881 (+ 215us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:27.944898 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:00:27.944956 (+ 58us) client_negotiation.cc:241] Negotiation successful
0430 08:00:27.945007 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260430 08:00:27.945676 16053 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:27.941544 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:43923 (local address 127.13.94.129:37421)
0430 08:00:27.941936 (+ 392us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:27.941941 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:27.941962 (+ 21us) server_negotiation.cc:408] Connection header received
0430 08:00:27.942116 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:27.942121 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:27.942256 (+ 135us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:27.942948 (+ 692us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:27.943107 (+ 159us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.943961 (+ 854us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:27.944853 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.945358 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:27.945412 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:27.945472 (+ 60us) server_negotiation.cc:300] Negotiation successful
0430 08:00:27.945529 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":138,"threads_started":1}
I20260430 08:00:27.945735 16054 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:27.941648 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:36749 (local address 127.13.94.131:37917)
0430 08:00:27.942387 (+ 739us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:27.942394 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:27.942516 (+ 122us) server_negotiation.cc:408] Connection header received
0430 08:00:27.942652 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:27.942658 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:27.942720 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:27.942812 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:27.943340 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.943846 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:27.944822 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.945372 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:27.945413 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:27.945483 (+ 70us) server_negotiation.cc:300] Negotiation successful
0430 08:00:27.945545 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":97,"threads_started":1}
I20260430 08:00:27.946164 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e15566f01b542bdb88d63f25d6e75b3" candidate_uuid: "5bd2f44156784407af1156550cdfacf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a" is_pre_election: true
I20260430 08:00:27.946210 15892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e15566f01b542bdb88d63f25d6e75b3" candidate_uuid: "5bd2f44156784407af1156550cdfacf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6741a14d3a1427db23f3d7e537d6def" is_pre_election: true
I20260430 08:00:27.946413 15608 raft_consensus.cc:2468] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bd2f44156784407af1156550cdfacf4 in term 0.
I20260430 08:00:27.946511 15892 raft_consensus.cc:2468] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bd2f44156784407af1156550cdfacf4 in term 0.
I20260430 08:00:27.946887 15682 leader_election.cc:304] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd2f44156784407af1156550cdfacf4, 96467eab96b84e8c91246a631e9fdc4a; no voters:
I20260430 08:00:27.947193 16050 raft_consensus.cc:2804] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:27.947263 16050 raft_consensus.cc:493] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:27.947300 16050 raft_consensus.cc:3060] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:27.948226 16050 raft_consensus.cc:515] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:27.948596 16050 leader_election.cc:290] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [CANDIDATE]: Term 1 election: Requested vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), d6741a14d3a1427db23f3d7e537d6def (127.13.94.131:37917)
I20260430 08:00:27.949028 15892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e15566f01b542bdb88d63f25d6e75b3" candidate_uuid: "5bd2f44156784407af1156550cdfacf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6741a14d3a1427db23f3d7e537d6def"
I20260430 08:00:27.949018 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e15566f01b542bdb88d63f25d6e75b3" candidate_uuid: "5bd2f44156784407af1156550cdfacf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a"
I20260430 08:00:27.949136 15892 raft_consensus.cc:3060] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:27.949218 15608 raft_consensus.cc:3060] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:27.950197 15608 raft_consensus.cc:2468] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bd2f44156784407af1156550cdfacf4 in term 1.
I20260430 08:00:27.950198 15892 raft_consensus.cc:2468] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bd2f44156784407af1156550cdfacf4 in term 1.
I20260430 08:00:27.950595 15683 leader_election.cc:304] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd2f44156784407af1156550cdfacf4, d6741a14d3a1427db23f3d7e537d6def; no voters:
I20260430 08:00:27.950800 16050 raft_consensus.cc:2804] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:27.951053 16050 raft_consensus.cc:697] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 1 LEADER]: Becoming Leader. State: Replica: 5bd2f44156784407af1156550cdfacf4, State: Running, Role: LEADER
I20260430 08:00:27.951334 16050 consensus_queue.cc:237] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:27.953722 15448 catalog_manager.cc:5671] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bd2f44156784407af1156550cdfacf4 (127.13.94.130). New cstate: current_term: 1 leader_uuid: "5bd2f44156784407af1156550cdfacf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } health_report { overall_health: UNKNOWN } } }
I20260430 08:00:27.980891 16059 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:27.977761 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.13.94.130:37527)
0430 08:00:27.978004 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:27.978007 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:27.978019 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:00:27.978131 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:27.978134 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:27.978184 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:27.978282 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:27.978597 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.979167 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:27.979860 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.980032 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:27.980129 (+ 97us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:27.980505 (+ 376us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:27.980593 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:27.980670 (+ 77us) server_negotiation.cc:300] Negotiation successful
0430 08:00:27.980773 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":109,"threads_started":1}
I20260430 08:00:27.990541 16054 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:27.987939 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:36715 (local address 127.13.94.131:37917)
0430 08:00:27.988076 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:27.988080 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:27.988260 (+ 180us) server_negotiation.cc:408] Connection header received
0430 08:00:27.988372 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:27.988374 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:27.988422 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:27.988490 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:27.988904 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.989387 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:27.990000 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.990143 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:27.990178 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:27.990244 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:27.990304 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:27.990408 (+ 104us) server_negotiation.cc:300] Negotiation successful
0430 08:00:27.990441 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 08:00:27.990556 16062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:27.987889 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:37917 (local address 127.13.94.130:36715)
0430 08:00:27.988183 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:27.988196 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:27.988286 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:27.988507 (+ 221us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:27.988509 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:27.988516 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:00:27.988811 (+ 295us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:27.988817 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.989477 (+ 660us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:27.989479 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:27.989909 (+ 430us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:27.989914 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:27.990001 (+ 87us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:27.990033 (+ 32us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:00:27.990326 (+ 293us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:00:27.990329 (+ 3us) client_negotiation.cc:770] Sending connection context
0430 08:00:27.990381 (+ 52us) client_negotiation.cc:241] Negotiation successful
0430 08:00:27.990436 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":108,"threads_started":1}
I20260430 08:00:27.996572 15608 raft_consensus.cc:1275] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Refusing update from remote peer 5bd2f44156784407af1156550cdfacf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:27.996629 15892 raft_consensus.cc:1275] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Refusing update from remote peer 5bd2f44156784407af1156550cdfacf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:27.997172 16056 consensus_queue.cc:1048] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:27.997366 16050 consensus_queue.cc:1048] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:28.013883 16066 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:28.010506 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:37527 (local address 127.13.94.131:50465)
0430 08:00:28.011120 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:28.011137 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:28.011248 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:28.011519 (+ 271us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:28.011526 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:28.011537 (+ 11us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:00:28.011766 (+ 229us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:28.011772 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:28.012488 (+ 716us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:28.012490 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:28.012981 (+ 491us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:28.012989 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:28.013109 (+ 120us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:28.013131 (+ 22us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:00:28.013544 (+ 413us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:00:28.013552 (+ 8us) client_negotiation.cc:770] Sending connection context
0430 08:00:28.013675 (+ 123us) client_negotiation.cc:241] Negotiation successful
0430 08:00:28.013738 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":125,"threads_started":1}
I20260430 08:00:28.013872 16059 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:28.010579 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:50465 (local address 127.13.94.130:37527)
0430 08:00:28.010723 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:28.010727 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:28.011222 (+ 495us) server_negotiation.cc:408] Connection header received
0430 08:00:28.011330 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:28.011333 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:28.011413 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:28.011485 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:28.011867 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:28.012370 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:28.013119 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:28.013315 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:28.013358 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:28.013439 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:28.013528 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:28.013690 (+ 162us) server_negotiation.cc:300] Negotiation successful
0430 08:00:28.013733 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260430 08:00:28.036926 15448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:28.037077 15448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:28.039152 15448 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa: Sending DeleteTablet for 3 replicas of tablet 4e15566f01b542bdb88d63f25d6e75b3
I20260430 08:00:28.039852 15588 tablet_service.cc:1558] Processing DeleteTablet for tablet 4e15566f01b542bdb88d63f25d6e75b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:28 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:58660
I20260430 08:00:28.039876 15730 tablet_service.cc:1558] Processing DeleteTablet for tablet 4e15566f01b542bdb88d63f25d6e75b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:28 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42174
I20260430 08:00:28.039928 15872 tablet_service.cc:1558] Processing DeleteTablet for tablet 4e15566f01b542bdb88d63f25d6e75b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:28 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:55906
I20260430 08:00:28.040067 16033 tablet_replica.cc:333] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: stopping tablet replica
I20260430 08:00:28.040058 16034 tablet_replica.cc:333] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: stopping tablet replica
I20260430 08:00:28.040181 16034 raft_consensus.cc:2243] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:28.040186 16033 raft_consensus.cc:2243] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:00:28.040248 16035 tablet_replica.cc:333] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: stopping tablet replica
I20260430 08:00:28.040284 16034 raft_consensus.cc:2272] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:28.040341 16035 raft_consensus.cc:2243] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:28.040441 16035 raft_consensus.cc:2272] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:28.040473 16033 raft_consensus.cc:2272] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:28.041046 16034 ts_tablet_manager.cc:1916] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:28.041194 16033 ts_tablet_manager.cc:1916] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:28.041695 15448 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:00:28.042091 15448 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:28.043411 16034 ts_tablet_manager.cc:1929] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:00:28.043478 16034 log.cc:1199] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/4e15566f01b542bdb88d63f25d6e75b3
I20260430 08:00:28.043499 16033 ts_tablet_manager.cc:1929] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:00:28.043563 16033 log.cc:1199] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/4e15566f01b542bdb88d63f25d6e75b3
I20260430 08:00:28.044284 16033 ts_tablet_manager.cc:1950] T 4e15566f01b542bdb88d63f25d6e75b3 P 5bd2f44156784407af1156550cdfacf4: Deleting consensus metadata
I20260430 08:00:28.044986 16034 ts_tablet_manager.cc:1950] T 4e15566f01b542bdb88d63f25d6e75b3 P 96467eab96b84e8c91246a631e9fdc4a: Deleting consensus metadata
I20260430 08:00:28.045094 15434 catalog_manager.cc:5002] TS 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527): tablet 4e15566f01b542bdb88d63f25d6e75b3 (table test-table [id=8db4943256e94d8ea558edf5081bb790]) successfully deleted
I20260430 08:00:28.045518 16035 ts_tablet_manager.cc:1916] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:28.047269 15730 tablet_service.cc:1511] Processing CreateTablet for tablet 663c06f657eb440e9ad3681bc31ab5ed (DEFAULT_TABLE table=test-table [id=3c23da5414c44340ac36543d793be4e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:28.047533 15730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663c06f657eb440e9ad3681bc31ab5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:28.047526 15872 tablet_service.cc:1511] Processing CreateTablet for tablet 663c06f657eb440e9ad3681bc31ab5ed (DEFAULT_TABLE table=test-table [id=3c23da5414c44340ac36543d793be4e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:28.047745 15872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663c06f657eb440e9ad3681bc31ab5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:28.047888 16035 ts_tablet_manager.cc:1929] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:00:28.047953 16035 log.cc:1199] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/4e15566f01b542bdb88d63f25d6e75b3
I20260430 08:00:28.048215 16035 ts_tablet_manager.cc:1950] T 4e15566f01b542bdb88d63f25d6e75b3 P d6741a14d3a1427db23f3d7e537d6def: Deleting consensus metadata
I20260430 08:00:28.048803 15588 tablet_service.cc:1511] Processing CreateTablet for tablet 663c06f657eb440e9ad3681bc31ab5ed (DEFAULT_TABLE table=test-table [id=3c23da5414c44340ac36543d793be4e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:28.049046 15588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 663c06f657eb440e9ad3681bc31ab5ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:28.049783 16046 tablet_bootstrap.cc:492] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: Bootstrap starting.
I20260430 08:00:28.050617 16044 tablet_bootstrap.cc:492] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: Bootstrap starting.
I20260430 08:00:28.050869 16046 tablet_bootstrap.cc:654] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:28.051229 16045 tablet_bootstrap.cc:492] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a: Bootstrap starting.
I20260430 08:00:28.051538 16044 tablet_bootstrap.cc:654] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:28.052052 16046 tablet_bootstrap.cc:492] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: No bootstrap required, opened a new log
I20260430 08:00:28.052121 16046 ts_tablet_manager.cc:1403] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:28.052147 16045 tablet_bootstrap.cc:654] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:28.052634 15434 catalog_manager.cc:5002] TS 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421): tablet 4e15566f01b542bdb88d63f25d6e75b3 (table test-table [id=8db4943256e94d8ea558edf5081bb790]) successfully deleted
I20260430 08:00:28.052677 16046 raft_consensus.cc:359] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:28.052798 16046 raft_consensus.cc:385] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:28.052829 16046 raft_consensus.cc:740] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd2f44156784407af1156550cdfacf4, State: Initialized, Role: FOLLOWER
I20260430 08:00:28.052955 16046 consensus_queue.cc:260] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:28.053229 16046 ts_tablet_manager.cc:1434] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260430 08:00:28.053293 16045 tablet_bootstrap.cc:492] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a: No bootstrap required, opened a new log
I20260430 08:00:28.053369 16045 ts_tablet_manager.cc:1403] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:28.053574 15435 catalog_manager.cc:5002] TS d6741a14d3a1427db23f3d7e537d6def (127.13.94.131:37917): tablet 4e15566f01b542bdb88d63f25d6e75b3 (table test-table [id=8db4943256e94d8ea558edf5081bb790]) successfully deleted
I20260430 08:00:28.053844 16045 raft_consensus.cc:359] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:28.053938 16045 raft_consensus.cc:385] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:28.053969 16045 raft_consensus.cc:740] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96467eab96b84e8c91246a631e9fdc4a, State: Initialized, Role: FOLLOWER
I20260430 08:00:28.054095 16045 consensus_queue.cc:260] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:28.054382 16045 ts_tablet_manager.cc:1434] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:28.054492 16044 tablet_bootstrap.cc:492] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: No bootstrap required, opened a new log
I20260430 08:00:28.054605 16044 ts_tablet_manager.cc:1403] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260430 08:00:28.055013 16044 raft_consensus.cc:359] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:28.055111 16044 raft_consensus.cc:385] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:28.055161 16044 raft_consensus.cc:740] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6741a14d3a1427db23f3d7e537d6def, State: Initialized, Role: FOLLOWER
I20260430 08:00:28.055289 16044 consensus_queue.cc:260] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:28.055553 16044 ts_tablet_manager.cc:1434] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:00:28.081606 16067 raft_consensus.cc:493] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:28.081764 16067 raft_consensus.cc:515] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:28.082242 16067 leader_election.cc:290] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), d6741a14d3a1427db23f3d7e537d6def (127.13.94.131:37917)
I20260430 08:00:28.082626 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "663c06f657eb440e9ad3681bc31ab5ed" candidate_uuid: "5bd2f44156784407af1156550cdfacf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a" is_pre_election: true
I20260430 08:00:28.082744 15892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "663c06f657eb440e9ad3681bc31ab5ed" candidate_uuid: "5bd2f44156784407af1156550cdfacf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6741a14d3a1427db23f3d7e537d6def" is_pre_election: true
I20260430 08:00:28.082779 15608 raft_consensus.cc:2468] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bd2f44156784407af1156550cdfacf4 in term 0.
I20260430 08:00:28.082885 15892 raft_consensus.cc:2468] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bd2f44156784407af1156550cdfacf4 in term 0.
I20260430 08:00:28.083134 15683 leader_election.cc:304] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd2f44156784407af1156550cdfacf4, d6741a14d3a1427db23f3d7e537d6def; no voters:
I20260430 08:00:28.083370 16067 raft_consensus.cc:2804] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:28.083429 16067 raft_consensus.cc:493] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:28.083467 16067 raft_consensus.cc:3060] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:28.084756 16067 raft_consensus.cc:515] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:28.085140 16067 leader_election.cc:290] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [CANDIDATE]: Term 1 election: Requested vote from peers 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421), d6741a14d3a1427db23f3d7e537d6def (127.13.94.131:37917)
I20260430 08:00:28.085455 15608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "663c06f657eb440e9ad3681bc31ab5ed" candidate_uuid: "5bd2f44156784407af1156550cdfacf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96467eab96b84e8c91246a631e9fdc4a"
I20260430 08:00:28.085588 15608 raft_consensus.cc:3060] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:28.086428 15892 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "663c06f657eb440e9ad3681bc31ab5ed" candidate_uuid: "5bd2f44156784407af1156550cdfacf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6741a14d3a1427db23f3d7e537d6def"
I20260430 08:00:28.086535 15892 raft_consensus.cc:3060] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:28.086614 15608 raft_consensus.cc:2468] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bd2f44156784407af1156550cdfacf4 in term 1.
I20260430 08:00:28.086900 15682 leader_election.cc:304] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd2f44156784407af1156550cdfacf4, 96467eab96b84e8c91246a631e9fdc4a; no voters:
I20260430 08:00:28.087046 16067 raft_consensus.cc:2804] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:28.087109 16067 raft_consensus.cc:697] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 1 LEADER]: Becoming Leader. State: Replica: 5bd2f44156784407af1156550cdfacf4, State: Running, Role: LEADER
I20260430 08:00:28.087258 16067 consensus_queue.cc:237] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } }
I20260430 08:00:28.087610 15892 raft_consensus.cc:2468] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bd2f44156784407af1156550cdfacf4 in term 1.
I20260430 08:00:28.088888 15447 catalog_manager.cc:5671] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bd2f44156784407af1156550cdfacf4 (127.13.94.130). New cstate: current_term: 1 leader_uuid: "5bd2f44156784407af1156550cdfacf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bd2f44156784407af1156550cdfacf4" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 } health_report { overall_health: UNKNOWN } } }
I20260430 08:00:28.108824 15608 raft_consensus.cc:1275] T 663c06f657eb440e9ad3681bc31ab5ed P 96467eab96b84e8c91246a631e9fdc4a [term 1 FOLLOWER]: Refusing update from remote peer 5bd2f44156784407af1156550cdfacf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:28.108798 15892 raft_consensus.cc:1275] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Refusing update from remote peer 5bd2f44156784407af1156550cdfacf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:28.109194 16067 consensus_queue.cc:1048] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96467eab96b84e8c91246a631e9fdc4a" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:28.109361 16050 consensus_queue.cc:1048] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6741a14d3a1427db23f3d7e537d6def" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260430 08:00:28.125293 16016 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260430 08:00:29.678171 15448 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:29.678339 15448 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:29.680613 15448 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4cf9e3981063446dae66a6d8efec9caa: Sending DeleteTablet for 3 replicas of tablet 663c06f657eb440e9ad3681bc31ab5ed
I20260430 08:00:29.681186 15588 tablet_service.cc:1558] Processing DeleteTablet for tablet 663c06f657eb440e9ad3681bc31ab5ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:29 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:58660
I20260430 08:00:29.681345 15872 tablet_service.cc:1558] Processing DeleteTablet for tablet 663c06f657eb440e9ad3681bc31ab5ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:29 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:55906
I20260430 08:00:29.681365 15730 tablet_service.cc:1558] Processing DeleteTablet for tablet 663c06f657eb440e9ad3681bc31ab5ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:29 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42174
I20260430 08:00:29.681684 16098 tablet_replica.cc:333] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: stopping tablet replica
I20260430 08:00:29.681690 16099 tablet_replica.cc:333] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: stopping tablet replica
I20260430 08:00:29.681785 16099 raft_consensus.cc:2243] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:00:29.681855 16098 raft_consensus.cc:2243] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:29.682005 16099 raft_consensus.cc:2272] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:29.682005 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 15523
I20260430 08:00:29.683205 16099 ts_tablet_manager.cc:1916] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:29.683559 16098 raft_consensus.cc:2272] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:29.684284 16098 ts_tablet_manager.cc:1916] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:29.685490 16099 ts_tablet_manager.cc:1929] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260430 08:00:29.685582 16099 log.cc:1199] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/663c06f657eb440e9ad3681bc31ab5ed
I20260430 08:00:29.685940 16099 ts_tablet_manager.cc:1950] T 663c06f657eb440e9ad3681bc31ab5ed P 5bd2f44156784407af1156550cdfacf4: Deleting consensus metadata
I20260430 08:00:29.686596 15434 catalog_manager.cc:5002] TS 5bd2f44156784407af1156550cdfacf4 (127.13.94.130:37527): tablet 663c06f657eb440e9ad3681bc31ab5ed (table test-table [id=3c23da5414c44340ac36543d793be4e0]) successfully deleted
I20260430 08:00:29.687330 16098 ts_tablet_manager.cc:1929] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260430 08:00:29.687412 16098 log.cc:1199] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/663c06f657eb440e9ad3681bc31ab5ed
I20260430 08:00:29.687731 16098 ts_tablet_manager.cc:1950] T 663c06f657eb440e9ad3681bc31ab5ed P d6741a14d3a1427db23f3d7e537d6def: Deleting consensus metadata
I20260430 08:00:29.688436 15435 catalog_manager.cc:5002] TS d6741a14d3a1427db23f3d7e537d6def (127.13.94.131:37917): tablet 663c06f657eb440e9ad3681bc31ab5ed (table test-table [id=3c23da5414c44340ac36543d793be4e0]) successfully deleted
W20260430 08:00:29.690841 15434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.94.129:37421 (error 108)
I20260430 08:00:29.691185 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 15665
I20260430 08:00:29.693632 16101 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:29.693231 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:37421 (local address 127.0.0.1:58670)
0430 08:00:29.693493 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:29.693548 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.129:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
W20260430 08:00:29.693892 15434 catalog_manager.cc:4729] TS 96467eab96b84e8c91246a631e9fdc4a (127.13.94.129:37421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 663c06f657eb440e9ad3681bc31ab5ed: Network error: Client connection negotiation failed: client connection to 127.13.94.129:37421: connect: Connection refused (error 111)
I20260430 08:00:29.698346 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 15807
I20260430 08:00:29.705260 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 15417
2026-04-30T08:00:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7701 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:29 dist-test-slave-9bxp krb5kdc[16104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:29 dist-test-slave-9bxp krb5kdc[16104](info): set up 2 sockets
Apr 30 08:00:29 dist-test-slave-9bxp krb5kdc[16104](info): commencing operation
krb5kdc: starting...
W20260430 08:00:31.721128 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.993s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:00:31 dist-test-slave-9bxp krb5kdc[16104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:31.864156 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:33797
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:33797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:31.971295 16120 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:31.971653 16120 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:31.971755 16120 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:31.975579 16120 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:31.975685 16120 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:31.975731 16120 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:31.975791 16120 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:31.975832 16120 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:31.980597 16120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:33797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:33797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:31.981866 16120 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:31.982806 16120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:31.988978 16125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:31.988996 16126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:31.989089 16120 server_base.cc:1061] running on GCE node
W20260430 08:00:31.988996 16128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:31.989766 16120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:31.990804 16120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:31.991997 16120 hybrid_clock.cc:648] HybridClock initialized: now 1777536031991969 us; error 40 us; skew 500 ppm
Apr 30 08:00:31 dist-test-slave-9bxp krb5kdc[16104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536031, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:31.994830 16120 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:31.995915 16120 webserver.cc:492] Webserver started at http://127.13.94.190:40859/ using document root <none> and password file <none>
I20260430 08:00:31.996464 16120 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:31.996515 16120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:31.996678 16120 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:31.998524 16120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "17df64dbc84245c5a9bdbc9af06a3895"
format_stamp: "Formatted at 2026-04-30 08:00:31 on dist-test-slave-9bxp"
server_key: "92d5348944c57bb677520b1f8ad6663a"
server_key_iv: "f6e2b14b092a57b1016491f63204ef81"
server_key_version: "encryptionkey@0"
I20260430 08:00:31.998983 16120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "17df64dbc84245c5a9bdbc9af06a3895"
format_stamp: "Formatted at 2026-04-30 08:00:31 on dist-test-slave-9bxp"
server_key: "92d5348944c57bb677520b1f8ad6663a"
server_key_iv: "f6e2b14b092a57b1016491f63204ef81"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.002614 16120 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:32.004909 16135 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.005967 16120 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:00:32.006116 16120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "17df64dbc84245c5a9bdbc9af06a3895"
format_stamp: "Formatted at 2026-04-30 08:00:31 on dist-test-slave-9bxp"
server_key: "92d5348944c57bb677520b1f8ad6663a"
server_key_iv: "f6e2b14b092a57b1016491f63204ef81"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.006261 16120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:32.027741 16120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:32.030929 16120 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:32.031152 16120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:32.038699 16120 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:33797
I20260430 08:00:32.038708 16187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:33797 every 8 connection(s)
I20260430 08:00:32.039832 16120 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:32.040326 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 16120
I20260430 08:00:32.040464 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:32.040822 13690 external_mini_cluster.cc:1468] Setting key b8ff1ea36eef519c5d782135a0fc4c10
I20260430 08:00:32.043164 16188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:32.049265 16188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895: Bootstrap starting.
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:32.051505 16188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:32.052214 16188 log.cc:826] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:32.054194 16188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895: No bootstrap required, opened a new log
I20260430 08:00:32.054823 16191 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:32.042175 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.13.94.190:33797)
0430 08:00:32.042578 (+ 403us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:32.042588 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:32.042621 (+ 33us) server_negotiation.cc:408] Connection header received
0430 08:00:32.043811 (+ 1190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:32.043833 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:32.044124 (+ 291us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:32.044415 (+ 291us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:32.045557 (+ 1142us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.046321 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:32.047164 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.047393 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:32.050148 (+ 2755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:32.050231 (+ 83us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:32.050244 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:32.050277 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:32.052175 (+ 1898us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:32.052656 (+ 481us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:32.052661 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:32.052665 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:32.052799 (+ 134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:32.053055 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:32.053058 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:32.053060 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:32.053415 (+ 355us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:32.053608 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:32.053923 (+ 315us) server_negotiation.cc:300] Negotiation successful
0430 08:00:32.054167 (+ 244us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":134,"threads_started":1}
I20260430 08:00:32.057204 16188 raft_consensus.cc:359] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df64dbc84245c5a9bdbc9af06a3895" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33797 } }
I20260430 08:00:32.057420 16188 raft_consensus.cc:385] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:32.057542 16188 raft_consensus.cc:740] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17df64dbc84245c5a9bdbc9af06a3895, State: Initialized, Role: FOLLOWER
I20260430 08:00:32.057955 16188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df64dbc84245c5a9bdbc9af06a3895" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33797 } }
I20260430 08:00:32.058099 16188 raft_consensus.cc:399] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:32.058182 16188 raft_consensus.cc:493] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:32.058302 16188 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:32.059815 16188 raft_consensus.cc:515] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df64dbc84245c5a9bdbc9af06a3895" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33797 } }
I20260430 08:00:32.060256 16188 leader_election.cc:304] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17df64dbc84245c5a9bdbc9af06a3895; no voters:
I20260430 08:00:32.060606 16188 leader_election.cc:290] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:32.060830 16193 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:32.061115 16193 raft_consensus.cc:697] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [term 1 LEADER]: Becoming Leader. State: Replica: 17df64dbc84245c5a9bdbc9af06a3895, State: Running, Role: LEADER
I20260430 08:00:32.061461 16193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df64dbc84245c5a9bdbc9af06a3895" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33797 } }
I20260430 08:00:32.061921 16188 sys_catalog.cc:565] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:32.063112 16195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17df64dbc84245c5a9bdbc9af06a3895. Latest consensus state: current_term: 1 leader_uuid: "17df64dbc84245c5a9bdbc9af06a3895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df64dbc84245c5a9bdbc9af06a3895" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33797 } } }
I20260430 08:00:32.063257 16195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:32.063639 16202 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:32.063638 16194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17df64dbc84245c5a9bdbc9af06a3895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df64dbc84245c5a9bdbc9af06a3895" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33797 } } }
I20260430 08:00:32.063788 16194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:32.067129 16202 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:32.073432 16202 catalog_manager.cc:1357] Generated new cluster ID: 7b1054e51a96456d942762d05a52424b
I20260430 08:00:32.073652 16202 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:32.083873 16202 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:32.085079 16202 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:32.103556 16202 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 17df64dbc84245c5a9bdbc9af06a3895: Generated new TSK 0
I20260430 08:00:32.104352 16202 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:32.166368 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:33797
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:32.276788 16216 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:32.277056 16216 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:32.277148 16216 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:32.280459 16216 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:32.280560 16216 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:32.280664 16216 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:32.285070 16216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:33797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:32.286187 16216 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:32.287066 16216 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:32.293705 16224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:32.293675 16222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:32.293784 16221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:32.294025 16216 server_base.cc:1061] running on GCE node
I20260430 08:00:32.294483 16216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:32.295032 16216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:32.296241 16216 hybrid_clock.cc:648] HybridClock initialized: now 1777536032296204 us; error 48 us; skew 500 ppm
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536032, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:32.299121 16216 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:00:32.300310 16216 webserver.cc:492] Webserver started at http://127.13.94.129:41735/ using document root <none> and password file <none>
I20260430 08:00:32.300978 16216 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:32.301054 16216 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:32.301285 16216 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:32.303282 16216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "98048fe0a1ea48a791d231608022fb92"
format_stamp: "Formatted at 2026-04-30 08:00:32 on dist-test-slave-9bxp"
server_key: "aa326685726407ed212cff9ffa42f49c"
server_key_iv: "abb662d95e5aa82d523b03e5e8c576cb"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.303822 16216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "98048fe0a1ea48a791d231608022fb92"
format_stamp: "Formatted at 2026-04-30 08:00:32 on dist-test-slave-9bxp"
server_key: "aa326685726407ed212cff9ffa42f49c"
server_key_iv: "abb662d95e5aa82d523b03e5e8c576cb"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.307399 16216 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260430 08:00:32.309805 16231 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.310997 16216 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:32.311133 16216 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "98048fe0a1ea48a791d231608022fb92"
format_stamp: "Formatted at 2026-04-30 08:00:32 on dist-test-slave-9bxp"
server_key: "aa326685726407ed212cff9ffa42f49c"
server_key_iv: "abb662d95e5aa82d523b03e5e8c576cb"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.311259 16216 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:32.324819 16216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:32.328450 16216 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:32.328689 16216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:32.329517 16216 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:32.330529 16216 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:32.330581 16216 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.330654 16216 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:32.330686 16216 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.340283 16216 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:38589
I20260430 08:00:32.340318 16344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:38589 every 8 connection(s)
I20260430 08:00:32.341423 16216 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:00:32.343145 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 16216
I20260430 08:00:32.343343 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:32.343637 13690 external_mini_cluster.cc:1468] Setting key 80184caf584e2dc70b06d5b5d068deb6
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536032, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
I20260430 08:00:32.355988 16191 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:32.343299 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:42261 (local address 127.13.94.190:33797)
0430 08:00:32.343471 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:32.343477 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:32.344357 (+ 880us) server_negotiation.cc:408] Connection header received
0430 08:00:32.345420 (+ 1063us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:32.345423 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:32.345468 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:32.345536 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:32.347743 (+ 2207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.348277 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:32.349124 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.349387 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:32.351674 (+ 2287us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:32.351703 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:32.351709 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:32.351751 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:32.353758 (+ 2007us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:32.354421 (+ 663us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:32.354427 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:32.354430 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:32.354504 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:32.354860 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:32.354862 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:32.354863 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:32.355022 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:32.355103 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:32.355724 (+ 621us) server_negotiation.cc:300] Negotiation successful
0430 08:00:32.355852 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260430 08:00:32.356791 16347 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:32.343689 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:33797 (local address 127.13.94.129:42261)
0430 08:00:32.344175 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:32.344243 (+ 68us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:32.345212 (+ 969us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:32.345712 (+ 500us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:32.345727 (+ 15us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:32.346322 (+ 595us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:32.347542 (+ 1220us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:32.347558 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.348426 (+ 868us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:32.348434 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:32.348960 (+ 526us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:32.348973 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.349204 (+ 231us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:32.349846 (+ 642us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:32.349869 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:32.351470 (+ 1601us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:32.353929 (+ 2459us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:32.353935 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:32.353948 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:32.354299 (+ 351us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:32.354628 (+ 329us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:32.354631 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:32.354633 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:32.354766 (+ 133us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:32.355174 (+ 408us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:32.355181 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:32.355467 (+ 286us) client_negotiation.cc:770] Sending connection context
0430 08:00:32.355671 (+ 204us) client_negotiation.cc:241] Negotiation successful
0430 08:00:32.355898 (+ 227us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
I20260430 08:00:32.358186 16345 heartbeater.cc:344] Connected to a master server at 127.13.94.190:33797
I20260430 08:00:32.358479 16345 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:32.359025 16345 heartbeater.cc:507] Master 127.13.94.190:33797 requested a full tablet report, sending...
I20260430 08:00:32.360981 16152 ts_manager.cc:194] Registered new tserver with Master: 98048fe0a1ea48a791d231608022fb92 (127.13.94.129:38589)
I20260430 08:00:32.362787 16152 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:42261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:32.404009 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:33797
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:32.511606 16352 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:32.511859 16352 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:32.511955 16352 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:32.515659 16352 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:32.515738 16352 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:32.515852 16352 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:32.520460 16352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:33797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:32.521736 16352 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:32.522712 16352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:32.529642 16360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:32.529654 16357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:32.529847 16352 server_base.cc:1061] running on GCE node
W20260430 08:00:32.529655 16358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:32.530285 16352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:32.530858 16352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:32.532063 16352 hybrid_clock.cc:648] HybridClock initialized: now 1777536032532037 us; error 40 us; skew 500 ppm
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536032, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:32.534869 16352 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:00:32.535943 16352 webserver.cc:492] Webserver started at http://127.13.94.130:40157/ using document root <none> and password file <none>
I20260430 08:00:32.536501 16352 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:32.536597 16352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:32.536864 16352 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:32.538617 16352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "33e717eae2d14781a4d7c4775f8beb5b"
format_stamp: "Formatted at 2026-04-30 08:00:32 on dist-test-slave-9bxp"
server_key: "88417a2b167fc279831a4ea2840837b7"
server_key_iv: "8daaf28e2636648c48d87ad1da636a7a"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.539134 16352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "33e717eae2d14781a4d7c4775f8beb5b"
format_stamp: "Formatted at 2026-04-30 08:00:32 on dist-test-slave-9bxp"
server_key: "88417a2b167fc279831a4ea2840837b7"
server_key_iv: "8daaf28e2636648c48d87ad1da636a7a"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.542687 16352 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:32.545068 16367 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.546094 16352 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:00:32.546236 16352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "33e717eae2d14781a4d7c4775f8beb5b"
format_stamp: "Formatted at 2026-04-30 08:00:32 on dist-test-slave-9bxp"
server_key: "88417a2b167fc279831a4ea2840837b7"
server_key_iv: "8daaf28e2636648c48d87ad1da636a7a"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.546347 16352 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:32.567013 16352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:32.570222 16352 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:32.570462 16352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:32.571107 16352 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:32.572083 16352 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:32.572157 16352 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.572220 16352 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:32.572273 16352 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.582391 16352 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:39661
I20260430 08:00:32.582422 16480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:39661 every 8 connection(s)
I20260430 08:00:32.583421 16352 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:00:32.590386 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 16352
I20260430 08:00:32.590478 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:32.590711 13690 external_mini_cluster.cc:1468] Setting key a26b50013c55e853a9306488ae221d9d
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536032, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
I20260430 08:00:32.599666 16191 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:32.585274 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:39631 (local address 127.13.94.190:33797)
0430 08:00:32.585431 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:32.585434 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:32.586168 (+ 734us) server_negotiation.cc:408] Connection header received
0430 08:00:32.587052 (+ 884us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:32.587056 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:32.587119 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:32.587228 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:32.588810 (+ 1582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.589327 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:32.589994 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.590202 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:32.592672 (+ 2470us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:32.592732 (+ 60us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:32.592738 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:32.592767 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:32.597379 (+ 4612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:32.598041 (+ 662us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:32.598044 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:32.598045 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:32.598088 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:32.598460 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:32.598462 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:32.598464 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:32.598619 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:32.598700 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:32.599376 (+ 676us) server_negotiation.cc:300] Negotiation successful
0430 08:00:32.599510 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260430 08:00:32.600353 16483 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:32.585613 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:33797 (local address 127.13.94.130:39631)
0430 08:00:32.586027 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:32.586060 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:32.586852 (+ 792us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:32.587391 (+ 539us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:32.587400 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:32.587777 (+ 377us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:32.588629 (+ 852us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:32.588642 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.589462 (+ 820us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:32.589471 (+ 9us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:32.589872 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:32.589878 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.590103 (+ 225us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:32.590814 (+ 711us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:32.590843 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:32.592522 (+ 1679us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:32.597526 (+ 5004us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:32.597534 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:32.597549 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:32.597935 (+ 386us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:32.598183 (+ 248us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:32.598190 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:32.598194 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:32.598357 (+ 163us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:32.598736 (+ 379us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:32.598742 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:32.599091 (+ 349us) client_negotiation.cc:770] Sending connection context
0430 08:00:32.599353 (+ 262us) client_negotiation.cc:241] Negotiation successful
0430 08:00:32.599635 (+ 282us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20260430 08:00:32.601678 16481 heartbeater.cc:344] Connected to a master server at 127.13.94.190:33797
I20260430 08:00:32.601935 16481 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
I20260430 08:00:32.602550 16481 heartbeater.cc:507] Master 127.13.94.190:33797 requested a full tablet report, sending...
I20260430 08:00:32.603914 16152 ts_manager.cc:194] Registered new tserver with Master: 33e717eae2d14781a4d7c4775f8beb5b (127.13.94.130:39661)
I20260430 08:00:32.604591 16152 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:39631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:32.645227 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:33797
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:32.748327 16488 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:32.748606 16488 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:32.748721 16488 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:32.752262 16488 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:32.752370 16488 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:32.752486 16488 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:32.756847 16488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:33797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:32.758057 16488 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:32.758949 16488 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:32.765249 16496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:32.765224 16493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:32.765341 16494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:32.765676 16488 server_base.cc:1061] running on GCE node
I20260430 08:00:32.766026 16488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:32.766618 16488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:32.767788 16488 hybrid_clock.cc:648] HybridClock initialized: now 1777536032767768 us; error 39 us; skew 500 ppm
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536032, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:32.770448 16488 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:00:32.771557 16488 webserver.cc:492] Webserver started at http://127.13.94.131:41167/ using document root <none> and password file <none>
I20260430 08:00:32.772156 16488 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:32.772230 16488 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:32.772432 16488 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:32.774185 16488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "840c56b9e9af4942b521a2f4bf547163"
format_stamp: "Formatted at 2026-04-30 08:00:32 on dist-test-slave-9bxp"
server_key: "35a45912c829fef144449145c0985248"
server_key_iv: "bb8fc39676dea9fdefd6e24d01657726"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.774686 16488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "840c56b9e9af4942b521a2f4bf547163"
format_stamp: "Formatted at 2026-04-30 08:00:32 on dist-test-slave-9bxp"
server_key: "35a45912c829fef144449145c0985248"
server_key_iv: "bb8fc39676dea9fdefd6e24d01657726"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.778013 16488 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 08:00:32.780138 16503 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.781214 16488 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:32.781354 16488 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "840c56b9e9af4942b521a2f4bf547163"
format_stamp: "Formatted at 2026-04-30 08:00:32 on dist-test-slave-9bxp"
server_key: "35a45912c829fef144449145c0985248"
server_key_iv: "bb8fc39676dea9fdefd6e24d01657726"
server_key_version: "encryptionkey@0"
I20260430 08:00:32.781461 16488 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:32.801880 16488 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:32.805605 16488 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:32.805827 16488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:32.806444 16488 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:32.807348 16488 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:32.807420 16488 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.807499 16488 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:32.807535 16488 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:32.817385 16488 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:36661
I20260430 08:00:32.817409 16616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:36661 every 8 connection(s)
I20260430 08:00:32.818372 16488 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:00:32.820981 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 16488
I20260430 08:00:32.821112 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:32.821445 13690 external_mini_cluster.cc:1468] Setting key 1f8e7338e203d4db6e6ebb6feab27862
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536032, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:32.831432 16191 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:32.820213 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:40055 (local address 127.13.94.190:33797)
0430 08:00:32.820423 (+ 210us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:32.820428 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:32.821158 (+ 730us) server_negotiation.cc:408] Connection header received
0430 08:00:32.822268 (+ 1110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:32.822271 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:32.822334 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:32.822439 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:32.824023 (+ 1584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.824559 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:32.825300 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.825481 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:32.827845 (+ 2364us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:32.827870 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:32.827872 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:32.827902 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:32.829445 (+ 1543us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:32.829981 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:32.829985 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:32.829987 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:32.830042 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:32.830357 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:32.830360 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:32.830362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:32.830521 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:32.830626 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:32.831119 (+ 493us) server_negotiation.cc:300] Negotiation successful
0430 08:00:32.831255 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260430 08:00:32.832126 16619 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:32.820482 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:33797 (local address 127.13.94.131:40055)
0430 08:00:32.820994 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:32.821042 (+ 48us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:32.822050 (+ 1008us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:32.822615 (+ 565us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:32.822624 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:32.823002 (+ 378us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:32.823873 (+ 871us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:32.823886 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.824691 (+ 805us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:32.824756 (+ 65us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:32.825167 (+ 411us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:32.825174 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:32.825418 (+ 244us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:32.826061 (+ 643us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:32.826081 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:32.827672 (+ 1591us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:32.829600 (+ 1928us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:32.829607 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:32.829621 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:32.829870 (+ 249us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:32.830147 (+ 277us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:32.830150 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:32.830152 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:32.830251 (+ 99us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:32.830624 (+ 373us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:32.830631 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:32.830874 (+ 243us) client_negotiation.cc:770] Sending connection context
0430 08:00:32.831087 (+ 213us) client_negotiation.cc:241] Negotiation successful
0430 08:00:32.831331 (+ 244us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":93,"threads_started":1}
I20260430 08:00:32.833333 16617 heartbeater.cc:344] Connected to a master server at 127.13.94.190:33797
I20260430 08:00:32.833578 16617 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:32.834167 16617 heartbeater.cc:507] Master 127.13.94.190:33797 requested a full tablet report, sending...
I20260430 08:00:32.835422 16151 ts_manager.cc:194] Registered new tserver with Master: 840c56b9e9af4942b521a2f4bf547163 (127.13.94.131:36661)
I20260430 08:00:32.836025 16151 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:40055
I20260430 08:00:32.847611 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260430 08:00:32.859934 16191 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:00:32.858507 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.13.94.190:33797)
0430 08:00:32.858674 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:32.858682 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:32.858809 (+ 127us) server_negotiation.cc:408] Connection header received
0430 08:00:32.858932 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:32.858936 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:32.858989 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:32.859067 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:32.859791 (+ 724us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36560: BlockingRecv error: recv got EOF from 127.0.0.1:36560 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 08:00:32.861335 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 16216
I20260430 08:00:32.866968 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 16352
I20260430 08:00:32.873518 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 16488
I20260430 08:00:32.883069 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 16120
2026-04-30T08:00:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3177 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16629](info): set up 2 sockets
Apr 30 08:00:32 dist-test-slave-9bxp krb5kdc[16629](info): commencing operation
krb5kdc: starting...
W20260430 08:00:35.025699 13690 mini_kdc.cc:121] Time spent starting KDC: real 2.120s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260430 08:00:35.351328 13690 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.325s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260430 08:00:35.545375 13690 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.194s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260430 08:00:35.823486 13690 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.278s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/test-user.keytab.
W20260430 08:00:36.666644 13690 mini_kdc.cc:290] Time spent creating keytab for test-user: real 0.843s user 0.000s sys 0.002s
Apr 30 08:00:36 dist-test-slave-9bxp krb5kdc[16629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
W20260430 08:00:37.466706 13690 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.13.94.190: real 0.778s user 0.002s sys 0.002s
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
W20260430 08:00:37.571427 13690 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.13.94.190: real 0.105s user 0.001s sys 0.002s
I20260430 08:00:37.572443 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:38809
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:37.680883 16645 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:37.681125 16645 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:37.681227 16645 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:37.685039 16645 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:37.685118 16645 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:37.685142 16645 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:37.685160 16645 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:37.685220 16645 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:37.689756 16645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:38809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:37.691061 16645 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:37.692000 16645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:37.697923 16651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:37.697935 16653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:37.697935 16650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:37.698009 16645 server_base.cc:1061] running on GCE node
I20260430 08:00:37.698689 16645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:37.699703 16645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:37.700865 16645 hybrid_clock.cc:648] HybridClock initialized: now 1777536037700842 us; error 35 us; skew 500 ppm
Apr 30 08:00:37 dist-test-slave-9bxp krb5kdc[16629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536037, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:37.703784 16645 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:37.705017 16645 webserver.cc:492] Webserver started at http://127.13.94.190:45401/ using document root <none> and password file <none>
I20260430 08:00:37.705607 16645 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:37.705693 16645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:37.705910 16645 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:37.707671 16645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "e248199d5af14374ae2afbd4dc84f385"
format_stamp: "Formatted at 2026-04-30 08:00:37 on dist-test-slave-9bxp"
server_key: "2c24d345caecb89888d4afb58b08a80e"
server_key_iv: "bf9b256f444e136771b1a23398408682"
server_key_version: "encryptionkey@0"
I20260430 08:00:37.708212 16645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "e248199d5af14374ae2afbd4dc84f385"
format_stamp: "Formatted at 2026-04-30 08:00:37 on dist-test-slave-9bxp"
server_key: "2c24d345caecb89888d4afb58b08a80e"
server_key_iv: "bf9b256f444e136771b1a23398408682"
server_key_version: "encryptionkey@0"
I20260430 08:00:37.711689 16645 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260430 08:00:37.713950 16660 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:37.715003 16645 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:37.715157 16645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "e248199d5af14374ae2afbd4dc84f385"
format_stamp: "Formatted at 2026-04-30 08:00:37 on dist-test-slave-9bxp"
server_key: "2c24d345caecb89888d4afb58b08a80e"
server_key_iv: "bf9b256f444e136771b1a23398408682"
server_key_version: "encryptionkey@0"
I20260430 08:00:37.715270 16645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:37.738407 16645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:37.745126 16645 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:37.745345 16645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:37.753290 16645 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:38809
I20260430 08:00:37.753305 16712 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:38809 every 8 connection(s)
I20260430 08:00:37.754377 16645 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:37.757440 16713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:37.758705 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 16645
I20260430 08:00:37.758857 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:37.759107 13690 external_mini_cluster.cc:1468] Setting key 060ef96fe0c692b2a2fe859fa1228224
I20260430 08:00:37.764596 16713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385: Bootstrap starting.
Apr 30 08:00:37 dist-test-slave-9bxp krb5kdc[16629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:37.767544 16713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:37.768299 16713 log.cc:826] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:37.770649 16713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385: No bootstrap required, opened a new log
I20260430 08:00:37.772769 16716 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:37.760513 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.13.94.190:38809)
0430 08:00:37.760963 (+ 450us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:37.760973 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:37.761007 (+ 34us) server_negotiation.cc:408] Connection header received
0430 08:00:37.761565 (+ 558us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:37.761582 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:37.761905 (+ 323us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:37.762180 (+ 275us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:37.763347 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:37.764086 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:37.764955 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:37.765154 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:37.767762 (+ 2608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:37.767780 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:37.767793 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:37.767821 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:37.769979 (+ 2158us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:37.770403 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:37.770408 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:37.770414 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:37.770504 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:37.770780 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:37.770783 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:37.770785 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:37.771214 (+ 429us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:37.771410 (+ 196us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:37.771722 (+ 312us) server_negotiation.cc:300] Negotiation successful
0430 08:00:37.771973 (+ 251us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":168,"threads_started":1}
I20260430 08:00:37.773921 16713 raft_consensus.cc:359] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e248199d5af14374ae2afbd4dc84f385" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38809 } }
I20260430 08:00:37.774106 16713 raft_consensus.cc:385] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:37.774169 16713 raft_consensus.cc:740] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e248199d5af14374ae2afbd4dc84f385, State: Initialized, Role: FOLLOWER
I20260430 08:00:37.774633 16713 consensus_queue.cc:260] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e248199d5af14374ae2afbd4dc84f385" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38809 } }
I20260430 08:00:37.774781 16713 raft_consensus.cc:399] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:37.774863 16713 raft_consensus.cc:493] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:37.774974 16713 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:37.775882 16713 raft_consensus.cc:515] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e248199d5af14374ae2afbd4dc84f385" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38809 } }
I20260430 08:00:37.776235 16713 leader_election.cc:304] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e248199d5af14374ae2afbd4dc84f385; no voters:
I20260430 08:00:37.776563 16713 leader_election.cc:290] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:37.776971 16718 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:37.777205 16718 raft_consensus.cc:697] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [term 1 LEADER]: Becoming Leader. State: Replica: e248199d5af14374ae2afbd4dc84f385, State: Running, Role: LEADER
I20260430 08:00:37.777493 16718 consensus_queue.cc:237] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e248199d5af14374ae2afbd4dc84f385" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38809 } }
I20260430 08:00:37.777801 16713 sys_catalog.cc:565] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:37.779157 16719 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e248199d5af14374ae2afbd4dc84f385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e248199d5af14374ae2afbd4dc84f385" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38809 } } }
I20260430 08:00:37.779196 16720 sys_catalog.cc:455] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e248199d5af14374ae2afbd4dc84f385. Latest consensus state: current_term: 1 leader_uuid: "e248199d5af14374ae2afbd4dc84f385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e248199d5af14374ae2afbd4dc84f385" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38809 } } }
I20260430 08:00:37.779305 16719 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:37.779330 16720 sys_catalog.cc:458] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:37.779623 16727 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:37.782923 16727 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:37.789366 16727 catalog_manager.cc:1357] Generated new cluster ID: f186b3bc93c548f39e578f31de467d35
I20260430 08:00:37.789446 16727 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:37.820112 16727 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:37.821221 16727 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:37.827771 16727 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e248199d5af14374ae2afbd4dc84f385: Generated new TSK 0
I20260430 08:00:37.828411 16727 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:37.895644 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:38809
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:38.015949 16741 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:38.016193 16741 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:38.016254 16741 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:38.019656 16741 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:38.019733 16741 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:38.019871 16741 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:38.024403 16741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:38809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:38.025552 16741 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:38.026458 16741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:38.033366 16749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:38.033412 16747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:38.033370 16746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:38.033716 16741 server_base.cc:1061] running on GCE node
I20260430 08:00:38.034068 16741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:38.034602 16741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:38.035784 16741 hybrid_clock.cc:648] HybridClock initialized: now 1777536038035769 us; error 26 us; skew 500 ppm
Apr 30 08:00:38 dist-test-slave-9bxp krb5kdc[16629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536038, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:38.038601 16741 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:00:38.039637 16741 webserver.cc:492] Webserver started at http://127.13.94.129:45047/ using document root <none> and password file <none>
I20260430 08:00:38.040215 16741 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:38.040289 16741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:38.040506 16741 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:38.042250 16741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "2bf55caded9a40edbf987bf50fc29d98"
format_stamp: "Formatted at 2026-04-30 08:00:38 on dist-test-slave-9bxp"
server_key: "0aadb4bb329aeabf295c71aedf696e80"
server_key_iv: "503690d0453c31d0106f7cc837ac4439"
server_key_version: "encryptionkey@0"
I20260430 08:00:38.042764 16741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "2bf55caded9a40edbf987bf50fc29d98"
format_stamp: "Formatted at 2026-04-30 08:00:38 on dist-test-slave-9bxp"
server_key: "0aadb4bb329aeabf295c71aedf696e80"
server_key_iv: "503690d0453c31d0106f7cc837ac4439"
server_key_version: "encryptionkey@0"
I20260430 08:00:38.046263 16741 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:38.048679 16756 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:38.049850 16741 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:38.049991 16741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "2bf55caded9a40edbf987bf50fc29d98"
format_stamp: "Formatted at 2026-04-30 08:00:38 on dist-test-slave-9bxp"
server_key: "0aadb4bb329aeabf295c71aedf696e80"
server_key_iv: "503690d0453c31d0106f7cc837ac4439"
server_key_version: "encryptionkey@0"
I20260430 08:00:38.050102 16741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:38.062445 16741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:38.065073 16741 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:38.065284 16741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:38.065855 16741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:38.066751 16741 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:38.066825 16741 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:38.066898 16741 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:38.066933 16741 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:38.076572 16741 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:46627
I20260430 08:00:38.076619 16869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:46627 every 8 connection(s)
I20260430 08:00:38.077673 16741 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:00:38.082533 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 16741
I20260430 08:00:38.082653 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:38.082952 13690 external_mini_cluster.cc:1468] Setting key 20879e9118b0c09503765b84f54344aa
Apr 30 08:00:38 dist-test-slave-9bxp krb5kdc[16629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536038, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:38.092542 16716 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:38.079604 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:59081 (local address 127.13.94.190:38809)
0430 08:00:38.079754 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:38.079758 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:38.080567 (+ 809us) server_negotiation.cc:408] Connection header received
0430 08:00:38.081460 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:38.081464 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:38.081523 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:38.081633 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:38.083119 (+ 1486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.083833 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:38.084788 (+ 955us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.084929 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:38.088183 (+ 3254us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:38.088212 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:38.088218 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:38.088257 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:38.090273 (+ 2016us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:38.090886 (+ 613us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:38.090892 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:38.090896 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:38.090963 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:38.091297 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:38.091302 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:38.091305 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:38.091533 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:38.091662 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:38.092175 (+ 513us) server_negotiation.cc:300] Negotiation successful
0430 08:00:38.092328 (+ 153us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
I20260430 08:00:38.093283 16872 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:38.079910 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:38809 (local address 127.13.94.129:59081)
0430 08:00:38.080432 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:38.080466 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:38.081254 (+ 788us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:38.081827 (+ 573us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:38.081837 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:38.082240 (+ 403us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:38.082945 (+ 705us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:38.082960 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.084004 (+ 1044us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:38.084008 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:38.084616 (+ 608us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:38.084628 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.084889 (+ 261us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:38.086248 (+ 1359us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:38.086280 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:38.087985 (+ 1705us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:38.090436 (+ 2451us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:38.090443 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:38.090456 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:38.090754 (+ 298us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:38.091088 (+ 334us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:38.091091 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:38.091093 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:38.091193 (+ 100us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:38.091661 (+ 468us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:38.091667 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:38.091907 (+ 240us) client_negotiation.cc:770] Sending connection context
0430 08:00:38.092092 (+ 185us) client_negotiation.cc:241] Negotiation successful
0430 08:00:38.092303 (+ 211us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":95,"threads_started":1}
I20260430 08:00:38.094549 16870 heartbeater.cc:344] Connected to a master server at 127.13.94.190:38809
I20260430 08:00:38.094828 16870 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:38.095408 16870 heartbeater.cc:507] Master 127.13.94.190:38809 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
I20260430 08:00:38.097136 16673 ts_manager.cc:194] Registered new tserver with Master: 2bf55caded9a40edbf987bf50fc29d98 (127.13.94.129:46627)
I20260430 08:00:38.098865 16673 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:59081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:38.137458 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:38809
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:38.241079 16877 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:38.241344 16877 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:38.241431 16877 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:38.244881 16877 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:38.244971 16877 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:38.245079 16877 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:38.249528 16877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:38809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:38.250669 16877 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:38.251573 16877 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:38.258509 16882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:38.258512 16883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:38.258512 16885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:38.258865 16877 server_base.cc:1061] running on GCE node
I20260430 08:00:38.259246 16877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:38.259804 16877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:38.260964 16877 hybrid_clock.cc:648] HybridClock initialized: now 1777536038260930 us; error 44 us; skew 500 ppm
Apr 30 08:00:38 dist-test-slave-9bxp krb5kdc[16629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536038, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:38.263983 16877 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:00:38.265178 16877 webserver.cc:492] Webserver started at http://127.13.94.130:43105/ using document root <none> and password file <none>
I20260430 08:00:38.265771 16877 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:38.265849 16877 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:38.266057 16877 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:38.267787 16877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "d5467914cfb4448f907ff705c4fe865a"
format_stamp: "Formatted at 2026-04-30 08:00:38 on dist-test-slave-9bxp"
server_key: "deb16f0a0e3d6fd7242aaead50ff2a5e"
server_key_iv: "3a6642e6c2830b48a4108d9bcf49670f"
server_key_version: "encryptionkey@0"
I20260430 08:00:38.268285 16877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "d5467914cfb4448f907ff705c4fe865a"
format_stamp: "Formatted at 2026-04-30 08:00:38 on dist-test-slave-9bxp"
server_key: "deb16f0a0e3d6fd7242aaead50ff2a5e"
server_key_iv: "3a6642e6c2830b48a4108d9bcf49670f"
server_key_version: "encryptionkey@0"
I20260430 08:00:38.271925 16877 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260430 08:00:38.274574 16892 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:38.275753 16877 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:38.275875 16877 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "d5467914cfb4448f907ff705c4fe865a"
format_stamp: "Formatted at 2026-04-30 08:00:38 on dist-test-slave-9bxp"
server_key: "deb16f0a0e3d6fd7242aaead50ff2a5e"
server_key_iv: "3a6642e6c2830b48a4108d9bcf49670f"
server_key_version: "encryptionkey@0"
I20260430 08:00:38.276082 16877 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:38.296854 16877 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:38.300097 16877 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:38.300386 16877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:38.301082 16877 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:38.302043 16877 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:38.302093 16877 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:38.302134 16877 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:38.302150 16877 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:38.311700 16877 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:43679
I20260430 08:00:38.311733 17005 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:43679 every 8 connection(s)
I20260430 08:00:38.312830 16877 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:00:38.313268 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 16877
I20260430 08:00:38.313374 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:38.313668 13690 external_mini_cluster.cc:1468] Setting key f49b4520241745fd0e0084877ad50074
Apr 30 08:00:38 dist-test-slave-9bxp krb5kdc[16629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536038, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
I20260430 08:00:38.326342 16716 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:38.314702 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:43713 (local address 127.13.94.190:38809)
0430 08:00:38.314859 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:38.314864 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:38.315631 (+ 767us) server_negotiation.cc:408] Connection header received
0430 08:00:38.316488 (+ 857us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:38.316491 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:38.316545 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:38.316623 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:38.317800 (+ 1177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.318579 (+ 779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:38.319391 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.319636 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:38.322459 (+ 2823us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:38.322483 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:38.322487 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:38.322518 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:38.324293 (+ 1775us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:38.324916 (+ 623us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:38.324919 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:38.324920 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:38.324970 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:38.325251 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:38.325254 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:38.325255 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:38.325420 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:38.325497 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:38.326063 (+ 566us) server_negotiation.cc:300] Negotiation successful
0430 08:00:38.326197 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
I20260430 08:00:38.327162 17008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:38.315046 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:38809 (local address 127.13.94.130:43713)
0430 08:00:38.315495 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:38.315528 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:38.316298 (+ 770us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:38.316802 (+ 504us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:38.316811 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:38.317183 (+ 372us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:38.317659 (+ 476us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:38.317670 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.318748 (+ 1078us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:38.318752 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:38.319247 (+ 495us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:38.319256 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.319438 (+ 182us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:38.320684 (+ 1246us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:38.320761 (+ 77us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:38.322284 (+ 1523us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:38.324419 (+ 2135us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:38.324425 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:38.324437 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:38.324776 (+ 339us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:38.325061 (+ 285us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:38.325063 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:38.325065 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:38.325161 (+ 96us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:38.325519 (+ 358us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:38.325524 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:38.325828 (+ 304us) client_negotiation.cc:770] Sending connection context
0430 08:00:38.326002 (+ 174us) client_negotiation.cc:241] Negotiation successful
0430 08:00:38.326200 (+ 198us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260430 08:00:38.328377 17006 heartbeater.cc:344] Connected to a master server at 127.13.94.190:38809
I20260430 08:00:38.328675 17006 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:38.329254 17006 heartbeater.cc:507] Master 127.13.94.190:38809 requested a full tablet report, sending...
I20260430 08:00:38.330461 16673 ts_manager.cc:194] Registered new tserver with Master: d5467914cfb4448f907ff705c4fe865a (127.13.94.130:43679)
I20260430 08:00:38.331151 16673 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:43713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:38.368805 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:38809
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:38.471722 17013 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:38.472034 17013 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:38.472128 17013 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:38.475678 17013 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:38.475782 17013 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:38.475898 17013 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:38.480436 17013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:38809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:38.481580 17013 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:38.482460 17013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:38.489154 17019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:38.489177 17018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:38.489166 17021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:38.489454 17013 server_base.cc:1061] running on GCE node
I20260430 08:00:38.489874 17013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:38.490370 17013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:38.491568 17013 hybrid_clock.cc:648] HybridClock initialized: now 1777536038491550 us; error 30 us; skew 500 ppm
Apr 30 08:00:38 dist-test-slave-9bxp krb5kdc[16629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536038, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:38.494511 17013 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:00:38.495682 17013 webserver.cc:492] Webserver started at http://127.13.94.131:33927/ using document root <none> and password file <none>
I20260430 08:00:38.496295 17013 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:38.496367 17013 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:38.496574 17013 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:38.498450 17013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "e4f7a7f50d3d41c5b2dbf03c2ea59946"
format_stamp: "Formatted at 2026-04-30 08:00:38 on dist-test-slave-9bxp"
server_key: "af2c0885b812e7b2cedab57f248229c9"
server_key_iv: "abec238cac6fdd04a4d10f7e8add02a0"
server_key_version: "encryptionkey@0"
I20260430 08:00:38.498955 17013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "e4f7a7f50d3d41c5b2dbf03c2ea59946"
format_stamp: "Formatted at 2026-04-30 08:00:38 on dist-test-slave-9bxp"
server_key: "af2c0885b812e7b2cedab57f248229c9"
server_key_iv: "abec238cac6fdd04a4d10f7e8add02a0"
server_key_version: "encryptionkey@0"
I20260430 08:00:38.502401 17013 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:38.504659 17028 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:38.505780 17013 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:38.505916 17013 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "e4f7a7f50d3d41c5b2dbf03c2ea59946"
format_stamp: "Formatted at 2026-04-30 08:00:38 on dist-test-slave-9bxp"
server_key: "af2c0885b812e7b2cedab57f248229c9"
server_key_iv: "abec238cac6fdd04a4d10f7e8add02a0"
server_key_version: "encryptionkey@0"
I20260430 08:00:38.506027 17013 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:38.527067 17013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:38.529772 17013 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:38.529986 17013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:38.530589 17013 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:38.531515 17013 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:38.531589 17013 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:38.531678 17013 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:38.531728 17013 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:38.541393 17013 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:37107
I20260430 08:00:38.541415 17141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:37107 every 8 connection(s)
I20260430 08:00:38.542436 17013 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:00:38.544492 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 17013
I20260430 08:00:38.544638 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:38.544978 13690 external_mini_cluster.cc:1468] Setting key 850622af9238cd98e4f09f550ea803e3
Apr 30 08:00:38 dist-test-slave-9bxp krb5kdc[16629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536038, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:38.555665 16716 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:38.544227 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:35279 (local address 127.13.94.190:38809)
0430 08:00:38.544375 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:38.544378 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:38.545337 (+ 959us) server_negotiation.cc:408] Connection header received
0430 08:00:38.546440 (+ 1103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:38.546444 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:38.546507 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:38.546600 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:38.548025 (+ 1425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.548632 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:38.549304 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.549452 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:38.552170 (+ 2718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:38.552191 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:38.552193 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:38.552223 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:38.553830 (+ 1607us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:38.554299 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:38.554301 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:38.554303 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:38.554347 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:38.554706 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:38.554709 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:38.554710 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:38.554867 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:38.554948 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:38.555429 (+ 481us) server_negotiation.cc:300] Negotiation successful
0430 08:00:38.555533 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260430 08:00:38.556358 17144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:38.544608 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:38809 (local address 127.13.94.131:35279)
0430 08:00:38.545168 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:38.545207 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:38.546172 (+ 965us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:38.546800 (+ 628us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:38.546813 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:38.547383 (+ 570us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:38.547881 (+ 498us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:38.547893 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.548813 (+ 920us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:38.548816 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:38.549206 (+ 390us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:38.549211 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:38.549369 (+ 158us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:38.550431 (+ 1062us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:38.550451 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:38.551985 (+ 1534us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:38.553950 (+ 1965us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:38.553956 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:38.553965 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:38.554207 (+ 242us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:38.554462 (+ 255us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:38.554468 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:38.554474 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:38.554613 (+ 139us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:38.554960 (+ 347us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:38.554965 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:38.555192 (+ 227us) client_negotiation.cc:770] Sending connection context
0430 08:00:38.555372 (+ 180us) client_negotiation.cc:241] Negotiation successful
0430 08:00:38.555591 (+ 219us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":126,"threads_started":1}
I20260430 08:00:38.557551 17142 heartbeater.cc:344] Connected to a master server at 127.13.94.190:38809
I20260430 08:00:38.557842 17142 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:38.558372 17142 heartbeater.cc:507] Master 127.13.94.190:38809 requested a full tablet report, sending...
I20260430 08:00:38.559453 16673 ts_manager.cc:194] Registered new tserver with Master: e4f7a7f50d3d41c5b2dbf03c2ea59946 (127.13.94.131:37107)
I20260430 08:00:38.560019 16673 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:35279
I20260430 08:00:38.571053 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260430 08:00:38.700803 16716 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:00:38.697293 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.13.94.190:38809)
0430 08:00:38.697445 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:38.697449 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:38.698402 (+ 953us) server_negotiation.cc:408] Connection header received
0430 08:00:38.699329 (+ 927us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:38.699334 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:38.699395 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:38.699508 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:38.700642 (+ 1134us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42006: BlockingRecv error: recv got EOF from 127.0.0.1:42006 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260430 08:00:38.702258 16716 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:00:38.701451 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.13.94.190:38809)
0430 08:00:38.701603 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:38.701606 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:38.701620 (+ 14us) server_negotiation.cc:408] Connection header received
0430 08:00:38.701678 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:38.701680 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:38.701721 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:38.701789 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:38.702141 (+ 352us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42022: BlockingRecv error: recv got EOF from 127.0.0.1:42022 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260430 08:00:38.703190 16716 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:00:38.702542 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.13.94.190:38809)
0430 08:00:38.702671 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:38.702674 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:38.702684 (+ 10us) server_negotiation.cc:408] Connection header received
0430 08:00:38.702712 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:38.702714 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:38.702748 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:38.702811 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:38.703086 (+ 275us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42034: BlockingRecv error: recv got EOF from 127.0.0.1:42034 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260430 08:00:38.707176 16716 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:00:38.706363 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.13.94.190:38809)
0430 08:00:38.706474 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:38.706478 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:38.706490 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:00:38.706582 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:38.706584 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:38.706618 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:38.706694 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:38.707112 (+ 418us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42048: BlockingRecv error: recv got EOF from 127.0.0.1:42048 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260430 08:00:38.765906 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 16741
I20260430 08:00:38.771843 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 16877
I20260430 08:00:38.777274 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 17013
I20260430 08:00:38.782827 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 16645
2026-04-30T08:00:38Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (5899 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:38 dist-test-slave-9bxp krb5kdc[17165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:38 dist-test-slave-9bxp krb5kdc[17165](info): set up 2 sockets
Apr 30 08:00:38 dist-test-slave-9bxp krb5kdc[17165](info): commencing operation
krb5kdc: starting...
W20260430 08:00:40.793396 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:00:40 dist-test-slave-9bxp krb5kdc[17165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:40.939841 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:45675
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:41.045027 17181 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:41.045269 17181 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:41.045322 17181 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:41.048781 17181 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:41.048847 17181 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:41.048870 17181 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:41.048889 17181 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:41.048908 17181 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:41.053428 17181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:45675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:41.054629 17181 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:41.055562 17181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:41.061286 17187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:41.061281 17189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:41.061281 17186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:41.061347 17181 server_base.cc:1061] running on GCE node
I20260430 08:00:41.062053 17181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:41.062997 17181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:41.064152 17181 hybrid_clock.cc:648] HybridClock initialized: now 1777536041064133 us; error 41 us; skew 500 ppm
Apr 30 08:00:41 dist-test-slave-9bxp krb5kdc[17165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536041, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:41.066826 17181 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:41.067905 17181 webserver.cc:492] Webserver started at http://127.13.94.190:40307/ using document root <none> and password file <none>
I20260430 08:00:41.068459 17181 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:41.068508 17181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:41.068765 17181 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:41.070480 17181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "bcc5112e44ca4023b2f858b3ce321d8f"
format_stamp: "Formatted at 2026-04-30 08:00:41 on dist-test-slave-9bxp"
server_key: "964d212b0524e06146e7dbe1137a40d1"
server_key_iv: "e50e5e34f1b2115d943915b6e8345079"
server_key_version: "encryptionkey@0"
I20260430 08:00:41.070988 17181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "bcc5112e44ca4023b2f858b3ce321d8f"
format_stamp: "Formatted at 2026-04-30 08:00:41 on dist-test-slave-9bxp"
server_key: "964d212b0524e06146e7dbe1137a40d1"
server_key_iv: "e50e5e34f1b2115d943915b6e8345079"
server_key_version: "encryptionkey@0"
I20260430 08:00:41.074386 17181 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:41.076534 17196 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:41.077685 17181 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:41.077840 17181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "bcc5112e44ca4023b2f858b3ce321d8f"
format_stamp: "Formatted at 2026-04-30 08:00:41 on dist-test-slave-9bxp"
server_key: "964d212b0524e06146e7dbe1137a40d1"
server_key_iv: "e50e5e34f1b2115d943915b6e8345079"
server_key_version: "encryptionkey@0"
I20260430 08:00:41.077960 17181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:41.093606 17181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:41.096345 17181 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:41.096565 17181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:41.103993 17181 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:45675
I20260430 08:00:41.104004 17248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:45675 every 8 connection(s)
I20260430 08:00:41.105080 17181 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:41.105577 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 17181
I20260430 08:00:41.105726 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:41.105964 13690 external_mini_cluster.cc:1468] Setting key bc670b012f0eca4b6ccdf1cb39506afb
I20260430 08:00:41.108788 17249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Apr 30 08:00:41 dist-test-slave-9bxp krb5kdc[17165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:41.114616 17249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f: Bootstrap starting.
I20260430 08:00:41.117156 17249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:41.117897 17249 log.cc:826] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:41.119946 17252 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:41.107213 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.13.94.190:45675)
0430 08:00:41.107627 (+ 414us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:41.107638 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:41.107677 (+ 39us) server_negotiation.cc:408] Connection header received
0430 08:00:41.108535 (+ 858us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:41.108570 (+ 35us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:41.109001 (+ 431us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:41.109363 (+ 362us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:41.110308 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:41.111458 (+ 1150us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:41.112168 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:41.112601 (+ 433us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:41.114883 (+ 2282us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:41.114946 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:41.114959 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:41.114992 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:41.117016 (+ 2024us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:41.117738 (+ 722us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:41.117744 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:41.117751 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:41.117830 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:41.118130 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:41.118142 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:41.118144 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:41.118555 (+ 411us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:41.118773 (+ 218us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:41.119107 (+ 334us) server_negotiation.cc:300] Negotiation successful
0430 08:00:41.119354 (+ 247us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260430 08:00:41.120030 17249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f: No bootstrap required, opened a new log
I20260430 08:00:41.122735 13690 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 08:00:41.123380 17249 raft_consensus.cc:359] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc5112e44ca4023b2f858b3ce321d8f" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45675 } }
I20260430 08:00:41.123567 17249 raft_consensus.cc:385] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:41.123631 17249 raft_consensus.cc:740] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc5112e44ca4023b2f858b3ce321d8f, State: Initialized, Role: FOLLOWER
I20260430 08:00:41.124071 17249 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc5112e44ca4023b2f858b3ce321d8f" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45675 } }
I20260430 08:00:41.124188 17249 raft_consensus.cc:399] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:41.124269 17249 raft_consensus.cc:493] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:41.124361 17249 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:41.125305 17249 raft_consensus.cc:515] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc5112e44ca4023b2f858b3ce321d8f" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45675 } }
I20260430 08:00:41.125649 17249 leader_election.cc:304] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcc5112e44ca4023b2f858b3ce321d8f; no voters:
I20260430 08:00:41.125975 17249 leader_election.cc:290] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:41.126073 17255 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:41.126302 17255 raft_consensus.cc:697] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [term 1 LEADER]: Becoming Leader. State: Replica: bcc5112e44ca4023b2f858b3ce321d8f, State: Running, Role: LEADER
I20260430 08:00:41.126703 17255 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc5112e44ca4023b2f858b3ce321d8f" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45675 } }
I20260430 08:00:41.127344 17249 sys_catalog.cc:565] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:41.128258 17257 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc5112e44ca4023b2f858b3ce321d8f. Latest consensus state: current_term: 1 leader_uuid: "bcc5112e44ca4023b2f858b3ce321d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc5112e44ca4023b2f858b3ce321d8f" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45675 } } }
I20260430 08:00:41.128377 17257 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:41.128715 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcc5112e44ca4023b2f858b3ce321d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc5112e44ca4023b2f858b3ce321d8f" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45675 } } }
I20260430 08:00:41.128801 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:41.129010 17264 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:41.131963 17264 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:41.142488 17264 catalog_manager.cc:1357] Generated new cluster ID: b2f2b795aefa423991ff363d5e9d6335
I20260430 08:00:41.142581 17264 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:41.146353 17252 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:41.141234 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.13.94.190:45675)
0430 08:00:41.141383 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:41.141387 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:41.141790 (+ 403us) server_negotiation.cc:408] Connection header received
0430 08:00:41.141961 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:41.141965 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:41.142023 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:41.142116 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:41.142935 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:41.143754 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:41.144553 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:41.144813 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:41.145408 (+ 595us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:41.145431 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:41.145434 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:41.145874 (+ 440us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:41.145907 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:41.145922 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:41.146030 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:41.146144 (+ 114us) server_negotiation.cc:300] Negotiation successful
0430 08:00:41.146198 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 08:00:41.179055 17264 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:41.179949 17264 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:41.188719 17264 catalog_manager.cc:6044] T 00000000000000000000000000000000 P bcc5112e44ca4023b2f858b3ce321d8f: Generated new TSK 0
I20260430 08:00:41.189337 17264 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 08:00:41.194180 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 17181
2026-04-30T08:00:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2409 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:41 dist-test-slave-9bxp krb5kdc[17282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:41 dist-test-slave-9bxp krb5kdc[17282](info): set up 2 sockets
Apr 30 08:00:41 dist-test-slave-9bxp krb5kdc[17282](info): commencing operation
krb5kdc: starting...
W20260430 08:00:43.201514 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.987s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:00:43 dist-test-slave-9bxp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:43.341835 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:38937
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:38937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:43.442678 17298 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:43.442915 17298 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:43.442971 17298 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:43.446290 17298 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:43.446359 17298 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:43.446389 17298 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:43.446409 17298 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:43.446427 17298 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:43.450764 17298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:38937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:38937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:43.451781 17298 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:43.452579 17298 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:43.458276 17304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:43.458276 17303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:43.458375 17306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:43.458724 17298 server_base.cc:1061] running on GCE node
I20260430 08:00:43.459249 17298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:43.460186 17298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:43.461340 17298 hybrid_clock.cc:648] HybridClock initialized: now 1777536043461307 us; error 52 us; skew 500 ppm
Apr 30 08:00:43 dist-test-slave-9bxp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536043, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:43.464334 17298 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:43.465533 17298 webserver.cc:492] Webserver started at http://127.13.94.190:40219/ using document root <none> and password file <none>
I20260430 08:00:43.466094 17298 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:43.466145 17298 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:43.466346 17298 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:43.468113 17298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "a3c81a73765b4d41911b849356e07c32"
format_stamp: "Formatted at 2026-04-30 08:00:43 on dist-test-slave-9bxp"
server_key: "ef2be1124b1fe8d528188a901889ed13"
server_key_iv: "bd90bae7615c0e355ada315c02bffea6"
server_key_version: "encryptionkey@0"
I20260430 08:00:43.468605 17298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "a3c81a73765b4d41911b849356e07c32"
format_stamp: "Formatted at 2026-04-30 08:00:43 on dist-test-slave-9bxp"
server_key: "ef2be1124b1fe8d528188a901889ed13"
server_key_iv: "bd90bae7615c0e355ada315c02bffea6"
server_key_version: "encryptionkey@0"
I20260430 08:00:43.471879 17298 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260430 08:00:43.474211 17313 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:43.475284 17298 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260430 08:00:43.475487 17298 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "a3c81a73765b4d41911b849356e07c32"
format_stamp: "Formatted at 2026-04-30 08:00:43 on dist-test-slave-9bxp"
server_key: "ef2be1124b1fe8d528188a901889ed13"
server_key_iv: "bd90bae7615c0e355ada315c02bffea6"
server_key_version: "encryptionkey@0"
I20260430 08:00:43.475605 17298 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:43.517938 17298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:43.525854 17298 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:43.526075 17298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:43.533181 17298 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:38937
I20260430 08:00:43.533187 17365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:38937 every 8 connection(s)
I20260430 08:00:43.534209 17298 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:43.537258 17366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:43.537911 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 17298
I20260430 08:00:43.538045 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:43.538312 13690 external_mini_cluster.cc:1468] Setting key c501cb386135c2ff0232a0ba32a3c739
I20260430 08:00:43.544590 17366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32: Bootstrap starting.
Apr 30 08:00:43 dist-test-slave-9bxp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:43.547299 17366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:43.548041 17366 log.cc:826] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:43.549991 17366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32: No bootstrap required, opened a new log
I20260430 08:00:43.551591 17369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:43.539975 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.13.94.190:38937)
0430 08:00:43.540430 (+ 455us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:43.540440 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:43.540473 (+ 33us) server_negotiation.cc:408] Connection header received
0430 08:00:43.541110 (+ 637us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:43.541130 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:43.541403 (+ 273us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:43.541670 (+ 267us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:43.542817 (+ 1147us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:43.543573 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:43.544399 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:43.544642 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:43.547123 (+ 2481us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:43.547141 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:43.547152 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:43.547181 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:43.549066 (+ 1885us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:43.549521 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:43.549526 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:43.549531 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:43.549604 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:43.549860 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:43.549864 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:43.549866 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:43.550238 (+ 372us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:43.550425 (+ 187us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:43.550759 (+ 334us) server_negotiation.cc:300] Negotiation successful
0430 08:00:43.550998 (+ 239us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":164,"threads_started":1}
I20260430 08:00:43.552692 17366 raft_consensus.cc:359] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c81a73765b4d41911b849356e07c32" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38937 } }
I20260430 08:00:43.552959 17366 raft_consensus.cc:385] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:43.553043 17366 raft_consensus.cc:740] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c81a73765b4d41911b849356e07c32, State: Initialized, Role: FOLLOWER
I20260430 08:00:43.553478 17366 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c81a73765b4d41911b849356e07c32" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38937 } }
I20260430 08:00:43.553633 17366 raft_consensus.cc:399] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:43.553742 17366 raft_consensus.cc:493] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:43.553853 17366 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:43.554710 17366 raft_consensus.cc:515] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c81a73765b4d41911b849356e07c32" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38937 } }
I20260430 08:00:43.555069 17366 leader_election.cc:304] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3c81a73765b4d41911b849356e07c32; no voters:
I20260430 08:00:43.555509 17366 leader_election.cc:290] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:43.555594 17371 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:43.555797 17371 raft_consensus.cc:697] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [term 1 LEADER]: Becoming Leader. State: Replica: a3c81a73765b4d41911b849356e07c32, State: Running, Role: LEADER
I20260430 08:00:43.556084 17371 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c81a73765b4d41911b849356e07c32" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38937 } }
I20260430 08:00:43.556810 17366 sys_catalog.cc:565] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:43.557339 17372 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3c81a73765b4d41911b849356e07c32. Latest consensus state: current_term: 1 leader_uuid: "a3c81a73765b4d41911b849356e07c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c81a73765b4d41911b849356e07c32" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38937 } } }
I20260430 08:00:43.557356 17373 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3c81a73765b4d41911b849356e07c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3c81a73765b4d41911b849356e07c32" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38937 } } }
I20260430 08:00:43.557499 17372 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:43.557571 17373 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:43.557984 17375 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:43.561282 17375 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:43.567008 17375 catalog_manager.cc:1357] Generated new cluster ID: 4e352db3bb0d477ba7d65df46af32ccb
I20260430 08:00:43.567103 17375 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:43.576757 17375 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:43.577749 17375 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:43.592598 17375 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a3c81a73765b4d41911b849356e07c32: Generated new TSK 0
I20260430 08:00:43.593325 17375 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:43.647799 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:38937
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:43.752497 17394 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:43.752822 17394 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:43.752923 17394 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:43.756235 17394 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:43.756292 17394 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:43.756371 17394 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:43.760900 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:43.761960 17394 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:43.762845 17394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:43.769450 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:43.769490 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:43.769471 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:43.770051 17394 server_base.cc:1061] running on GCE node
I20260430 08:00:43.770368 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:43.770869 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:43.772013 17394 hybrid_clock.cc:648] HybridClock initialized: now 1777536043771990 us; error 36 us; skew 500 ppm
Apr 30 08:00:43 dist-test-slave-9bxp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536043, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:43.775005 17394 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:00:43.776113 17394 webserver.cc:492] Webserver started at http://127.13.94.129:38477/ using document root <none> and password file <none>
I20260430 08:00:43.776680 17394 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:43.776808 17394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:43.777006 17394 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:43.778641 17394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "d0ccded3a4ca42e88c56e6c9c3f108ce"
format_stamp: "Formatted at 2026-04-30 08:00:43 on dist-test-slave-9bxp"
server_key: "1d8682acae0d8811341f0c2f97ed3ab2"
server_key_iv: "cf1bcb18147967c321313a1e36886e18"
server_key_version: "encryptionkey@0"
I20260430 08:00:43.779121 17394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "d0ccded3a4ca42e88c56e6c9c3f108ce"
format_stamp: "Formatted at 2026-04-30 08:00:43 on dist-test-slave-9bxp"
server_key: "1d8682acae0d8811341f0c2f97ed3ab2"
server_key_iv: "cf1bcb18147967c321313a1e36886e18"
server_key_version: "encryptionkey@0"
I20260430 08:00:43.782526 17394 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260430 08:00:43.784962 17409 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:43.786080 17394 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:00:43.786221 17394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "d0ccded3a4ca42e88c56e6c9c3f108ce"
format_stamp: "Formatted at 2026-04-30 08:00:43 on dist-test-slave-9bxp"
server_key: "1d8682acae0d8811341f0c2f97ed3ab2"
server_key_iv: "cf1bcb18147967c321313a1e36886e18"
server_key_version: "encryptionkey@0"
I20260430 08:00:43.786345 17394 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:43.819821 17394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:43.822639 17394 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:43.822872 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:43.823477 17394 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:43.824399 17394 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:43.824472 17394 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:43.824540 17394 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:43.824584 17394 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:43.833964 17394 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:43143
I20260430 08:00:43.834033 17522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:43143 every 8 connection(s)
I20260430 08:00:43.835026 17394 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
Apr 30 08:00:43 dist-test-slave-9bxp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536043, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:43.844310 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 17394
I20260430 08:00:43.844419 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:43.844779 13690 external_mini_cluster.cc:1468] Setting key 37aca8868427a23b1e352605bdc71098
I20260430 08:00:43.848109 17369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:43.837107 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:35873 (local address 127.13.94.190:38937)
0430 08:00:43.837251 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:43.837254 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:43.838025 (+ 771us) server_negotiation.cc:408] Connection header received
0430 08:00:43.838860 (+ 835us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:43.838863 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:43.838914 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:43.838993 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:43.840253 (+ 1260us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:43.840911 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:43.841535 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:43.841672 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:43.844590 (+ 2918us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:43.844617 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:43.844621 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:43.844655 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:43.846193 (+ 1538us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:43.846693 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:43.846699 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:43.846704 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:43.846760 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:43.847033 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:43.847039 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:43.847044 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:43.847239 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:43.847343 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:43.847808 (+ 465us) server_negotiation.cc:300] Negotiation successful
0430 08:00:43.847950 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260430 08:00:43.849004 17525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:43.837395 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:38937 (local address 127.13.94.129:35873)
0430 08:00:43.837880 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:43.837914 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:43.838642 (+ 728us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:43.839148 (+ 506us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:43.839155 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:43.839536 (+ 381us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:43.840023 (+ 487us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:43.840034 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:43.841036 (+ 1002us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:43.841040 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:43.841431 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:43.841436 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:43.841597 (+ 161us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:43.842744 (+ 1147us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:43.842769 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:43.844357 (+ 1588us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:43.846332 (+ 1975us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:43.846337 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:43.846347 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:43.846585 (+ 238us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:43.846855 (+ 270us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:43.846857 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:43.846859 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:43.846941 (+ 82us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:43.847339 (+ 398us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:43.847343 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:43.847570 (+ 227us) client_negotiation.cc:770] Sending connection context
0430 08:00:43.847738 (+ 168us) client_negotiation.cc:241] Negotiation successful
0430 08:00:43.848051 (+ 313us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":110,"threads_started":1}
I20260430 08:00:43.850301 17523 heartbeater.cc:344] Connected to a master server at 127.13.94.190:38937
I20260430 08:00:43.850555 17523 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:43.851083 17523 heartbeater.cc:507] Master 127.13.94.190:38937 requested a full tablet report, sending...
I20260430 08:00:43.852810 17330 ts_manager.cc:194] Registered new tserver with Master: d0ccded3a4ca42e88c56e6c9c3f108ce (127.13.94.129:43143)
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
I20260430 08:00:43.854493 17330 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:35873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:43.895555 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:38937
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:43.998296 17530 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:43.998540 17530 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:43.998601 17530 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:44.002120 17530 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:44.002183 17530 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:44.002324 17530 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:44.006628 17530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:44.007735 17530 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:44.008608 17530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:44.015780 17535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:44.015842 17536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:44.015781 17538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:44.015952 17530 server_base.cc:1061] running on GCE node
I20260430 08:00:44.016448 17530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:44.017046 17530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:44.018225 17530 hybrid_clock.cc:648] HybridClock initialized: now 1777536044018195 us; error 50 us; skew 500 ppm
Apr 30 08:00:44 dist-test-slave-9bxp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536044, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:44.021126 17530 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:00:44.022149 17530 webserver.cc:492] Webserver started at http://127.13.94.130:36775/ using document root <none> and password file <none>
I20260430 08:00:44.022682 17530 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:44.022732 17530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:44.022888 17530 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:44.024472 17530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "23d04ddbfca640cf8a07da1a218f5c15"
format_stamp: "Formatted at 2026-04-30 08:00:44 on dist-test-slave-9bxp"
server_key: "04f8f03dea7372fe069b8b6d7c002a8f"
server_key_iv: "c5ad7c8a5611e7be90ef4a9afea19b9a"
server_key_version: "encryptionkey@0"
I20260430 08:00:44.024998 17530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "23d04ddbfca640cf8a07da1a218f5c15"
format_stamp: "Formatted at 2026-04-30 08:00:44 on dist-test-slave-9bxp"
server_key: "04f8f03dea7372fe069b8b6d7c002a8f"
server_key_iv: "c5ad7c8a5611e7be90ef4a9afea19b9a"
server_key_version: "encryptionkey@0"
I20260430 08:00:44.028244 17530 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260430 08:00:44.030479 17545 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:44.031469 17530 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:44.031563 17530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "23d04ddbfca640cf8a07da1a218f5c15"
format_stamp: "Formatted at 2026-04-30 08:00:44 on dist-test-slave-9bxp"
server_key: "04f8f03dea7372fe069b8b6d7c002a8f"
server_key_iv: "c5ad7c8a5611e7be90ef4a9afea19b9a"
server_key_version: "encryptionkey@0"
I20260430 08:00:44.031637 17530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:44.053308 17530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:44.056044 17530 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:44.056260 17530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:44.056897 17530 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:44.058004 17530 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:44.058053 17530 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:44.058120 17530 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:44.058157 17530 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:44.067051 17530 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:44521
I20260430 08:00:44.067099 17658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:44521 every 8 connection(s)
I20260430 08:00:44.068024 17530 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:00:44.071696 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 17530
I20260430 08:00:44.071816 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:44.072078 13690 external_mini_cluster.cc:1468] Setting key 2ed2da17c05958d42cb1a147562a00a5
Apr 30 08:00:44 dist-test-slave-9bxp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536044, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:44.082257 17369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:44.069828 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:59377 (local address 127.13.94.190:38937)
0430 08:00:44.069974 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:44.069977 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:44.070609 (+ 632us) server_negotiation.cc:408] Connection header received
0430 08:00:44.071423 (+ 814us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:44.071426 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:44.071477 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:44.071565 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:44.073167 (+ 1602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.073934 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:44.074958 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.075142 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:44.078201 (+ 3059us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:44.078228 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:44.078233 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:44.078271 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:44.080064 (+ 1793us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:44.080598 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:44.080602 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:44.080606 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:44.080663 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:44.080988 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:44.080994 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:44.080996 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:44.081201 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:44.081313 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:44.081910 (+ 597us) server_negotiation.cc:300] Negotiation successful
0430 08:00:44.082036 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
I20260430 08:00:44.082854 17661 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:44.070052 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:38937 (local address 127.13.94.130:59377)
0430 08:00:44.070449 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:44.070482 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:44.071243 (+ 761us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:44.071771 (+ 528us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:44.071784 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:44.072353 (+ 569us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:44.073017 (+ 664us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:44.073031 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.074054 (+ 1023us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:44.074065 (+ 11us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:44.074806 (+ 741us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:44.074839 (+ 33us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.075068 (+ 229us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:44.076209 (+ 1141us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:44.076228 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:44.078030 (+ 1802us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:44.080211 (+ 2181us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:44.080221 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:44.080236 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:44.080496 (+ 260us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:44.080802 (+ 306us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:44.080805 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:44.080806 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:44.080894 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:44.081344 (+ 450us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:44.081349 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:44.081621 (+ 272us) client_negotiation.cc:770] Sending connection context
0430 08:00:44.081822 (+ 201us) client_negotiation.cc:241] Negotiation successful
0430 08:00:44.082028 (+ 206us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
I20260430 08:00:44.084110 17659 heartbeater.cc:344] Connected to a master server at 127.13.94.190:38937
I20260430 08:00:44.084370 17659 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:44.085050 17659 heartbeater.cc:507] Master 127.13.94.190:38937 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
I20260430 08:00:44.086334 17330 ts_manager.cc:194] Registered new tserver with Master: 23d04ddbfca640cf8a07da1a218f5c15 (127.13.94.130:44521)
I20260430 08:00:44.086910 17330 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:59377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:44.131831 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:38937
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:44.237185 17666 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:44.237488 17666 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:44.237596 17666 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:44.241248 17666 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:44.241387 17666 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:44.241523 17666 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:44.246491 17666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:38937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:44.247725 17666 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:44.248653 17666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:44.255715 17672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:44.255714 17671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:44.255801 17674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:44.256094 17666 server_base.cc:1061] running on GCE node
I20260430 08:00:44.256575 17666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:44.257278 17666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:44.258519 17666 hybrid_clock.cc:648] HybridClock initialized: now 1777536044258500 us; error 32 us; skew 500 ppm
Apr 30 08:00:44 dist-test-slave-9bxp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536044, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:44.261942 17666 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:00:44.263131 17666 webserver.cc:492] Webserver started at http://127.13.94.131:45639/ using document root <none> and password file <none>
I20260430 08:00:44.263767 17666 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:44.263849 17666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:44.264072 17666 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:44.265942 17666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "aaa3bec6610f4f89af99dff4a15812d1"
format_stamp: "Formatted at 2026-04-30 08:00:44 on dist-test-slave-9bxp"
server_key: "dfa760b4c8e890cd997e4142a3ede28a"
server_key_iv: "3b353703898e85384bbd7d5d3c4f052c"
server_key_version: "encryptionkey@0"
I20260430 08:00:44.266448 17666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "aaa3bec6610f4f89af99dff4a15812d1"
format_stamp: "Formatted at 2026-04-30 08:00:44 on dist-test-slave-9bxp"
server_key: "dfa760b4c8e890cd997e4142a3ede28a"
server_key_iv: "3b353703898e85384bbd7d5d3c4f052c"
server_key_version: "encryptionkey@0"
I20260430 08:00:44.270269 17666 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260430 08:00:44.272895 17681 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:44.273931 17666 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:44.274077 17666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "aaa3bec6610f4f89af99dff4a15812d1"
format_stamp: "Formatted at 2026-04-30 08:00:44 on dist-test-slave-9bxp"
server_key: "dfa760b4c8e890cd997e4142a3ede28a"
server_key_iv: "3b353703898e85384bbd7d5d3c4f052c"
server_key_version: "encryptionkey@0"
I20260430 08:00:44.274192 17666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:44.297643 17666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:44.300815 17666 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:44.301041 17666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:44.301664 17666 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:44.302600 17666 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:44.302685 17666 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:44.302770 17666 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:44.302826 17666 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:44.313288 17666 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:34153
I20260430 08:00:44.313302 17794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:34153 every 8 connection(s)
I20260430 08:00:44.314306 17666 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:00:44.318625 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 17666
I20260430 08:00:44.318781 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:44.319067 13690 external_mini_cluster.cc:1468] Setting key f58d4a9ee2c2bae7b3546b6889c7c8a0
Apr 30 08:00:44 dist-test-slave-9bxp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536044, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:44.327687 17369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:44.316221 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:49017 (local address 127.13.94.190:38937)
0430 08:00:44.316410 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:44.316415 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:44.317296 (+ 881us) server_negotiation.cc:408] Connection header received
0430 08:00:44.318202 (+ 906us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:44.318206 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:44.318260 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:44.318342 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:44.320220 (+ 1878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.320962 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:44.321799 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.321934 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:44.324624 (+ 2690us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:44.324640 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:44.324642 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:44.324666 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:44.326043 (+ 1377us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:44.326500 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:44.326503 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:44.326505 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:44.326545 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:44.326812 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:44.326814 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:44.326815 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:44.326951 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:44.327024 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:44.327457 (+ 433us) server_negotiation.cc:300] Negotiation successful
0430 08:00:44.327560 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260430 08:00:44.328385 17797 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:44.316587 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:38937 (local address 127.13.94.131:49017)
0430 08:00:44.317141 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:44.317176 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:44.317985 (+ 809us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:44.318562 (+ 577us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:44.318576 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:44.319204 (+ 628us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:44.319983 (+ 779us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:44.320003 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.321088 (+ 1085us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:44.321092 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:44.321673 (+ 581us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:44.321681 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.321848 (+ 167us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:44.322942 (+ 1094us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:44.322961 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:44.324470 (+ 1509us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:44.326160 (+ 1690us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:44.326165 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:44.326175 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:44.326413 (+ 238us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:44.326622 (+ 209us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:44.326624 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:44.326626 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:44.326714 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:44.327037 (+ 323us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:44.327040 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:44.327247 (+ 207us) client_negotiation.cc:770] Sending connection context
0430 08:00:44.327414 (+ 167us) client_negotiation.cc:241] Negotiation successful
0430 08:00:44.327631 (+ 217us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":108,"threads_started":1}
I20260430 08:00:44.329527 17795 heartbeater.cc:344] Connected to a master server at 127.13.94.190:38937
I20260430 08:00:44.329787 17795 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:44.330289 17795 heartbeater.cc:507] Master 127.13.94.190:38937 requested a full tablet report, sending...
I20260430 08:00:44.331327 17330 ts_manager.cc:194] Registered new tserver with Master: aaa3bec6610f4f89af99dff4a15812d1 (127.13.94.131:34153)
I20260430 08:00:44.331890 17330 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:49017
I20260430 08:00:44.333308 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 30 08:00:44 dist-test-slave-9bxp krb5kdc[17282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536044, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Apr 30 08:00:44 dist-test-slave-9bxp krb5kdc[17282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536044, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:44.356920 17369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:44.349247 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.13.94.190:38937)
0430 08:00:44.349422 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:44.349426 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:44.349506 (+ 80us) server_negotiation.cc:408] Connection header received
0430 08:00:44.349649 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:44.349652 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:44.349705 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:44.349804 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:44.350617 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.351115 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:44.351720 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:44.351854 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:44.354007 (+ 2153us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:44.354025 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:44.354027 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:44.354049 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:44.355479 (+ 1430us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:44.355859 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:44.355862 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:44.355864 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:44.355909 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:44.356108 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:44.356110 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:44.356111 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:44.356272 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:44.356374 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:44.356594 (+ 220us) server_negotiation.cc:300] Negotiation successful
0430 08:00:44.356687 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260430 08:00:44.357348 17330 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42610
I20260430 08:00:44.358234 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 17394
I20260430 08:00:44.364535 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 17530
I20260430 08:00:44.370038 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 17666
I20260430 08:00:44.375581 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 17298
2026-04-30T08:00:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3182 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:44 dist-test-slave-9bxp krb5kdc[17807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:44 dist-test-slave-9bxp krb5kdc[17807](info): set up 2 sockets
Apr 30 08:00:44 dist-test-slave-9bxp krb5kdc[17807](info): commencing operation
krb5kdc: starting...
W20260430 08:00:46.393337 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.995s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:00:46 dist-test-slave-9bxp krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:46.540340 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:36527
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:46.647259 17823 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:46.647562 17823 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:46.647660 17823 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:46.651253 17823 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:46.651365 17823 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:46.651413 17823 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:46.651485 17823 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:46.651527 17823 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:46.655997 17823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:36527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:46.657186 17823 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:46.658077 17823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:46.663880 17829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:46.663884 17831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:46.663913 17828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:46.664219 17823 server_base.cc:1061] running on GCE node
I20260430 08:00:46.664814 17823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:46.665722 17823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:46.666880 17823 hybrid_clock.cc:648] HybridClock initialized: now 1777536046666859 us; error 36 us; skew 500 ppm
Apr 30 08:00:46 dist-test-slave-9bxp krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:46.669623 17823 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:46.670629 17823 webserver.cc:492] Webserver started at http://127.13.94.190:40181/ using document root <none> and password file <none>
I20260430 08:00:46.671135 17823 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:46.671181 17823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:46.671332 17823 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:46.673066 17823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "76c9e8f4b490447caeb1e2edbc76ef6c"
format_stamp: "Formatted at 2026-04-30 08:00:46 on dist-test-slave-9bxp"
server_key: "4f826c45142ccc9279e8e496becc6ad7"
server_key_iv: "9d7e882d50f1beffc0b0a767112bd4a5"
server_key_version: "encryptionkey@0"
I20260430 08:00:46.673498 17823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "76c9e8f4b490447caeb1e2edbc76ef6c"
format_stamp: "Formatted at 2026-04-30 08:00:46 on dist-test-slave-9bxp"
server_key: "4f826c45142ccc9279e8e496becc6ad7"
server_key_iv: "9d7e882d50f1beffc0b0a767112bd4a5"
server_key_version: "encryptionkey@0"
I20260430 08:00:46.676875 17823 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260430 08:00:46.679085 17838 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:46.680202 17823 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:46.680366 17823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "76c9e8f4b490447caeb1e2edbc76ef6c"
format_stamp: "Formatted at 2026-04-30 08:00:46 on dist-test-slave-9bxp"
server_key: "4f826c45142ccc9279e8e496becc6ad7"
server_key_iv: "9d7e882d50f1beffc0b0a767112bd4a5"
server_key_version: "encryptionkey@0"
I20260430 08:00:46.680526 17823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:46.689751 17823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:46.692880 17823 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:46.693084 17823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:46.700572 17823 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:36527
I20260430 08:00:46.700591 17890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:36527 every 8 connection(s)
I20260430 08:00:46.701689 17823 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:46.704885 17891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:46.706204 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 17823
I20260430 08:00:46.706521 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:46.706815 13690 external_mini_cluster.cc:1468] Setting key 65a8466f3e06e6b853c2cebc94e640fd
I20260430 08:00:46.710994 17891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c: Bootstrap starting.
I20260430 08:00:46.713352 17891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:46.714112 17891 log.cc:826] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c: Log is configured to *not* fsync() on all Append() calls
Apr 30 08:00:46 dist-test-slave-9bxp krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:46.716598 17891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c: No bootstrap required, opened a new log
I20260430 08:00:46.719743 17891 raft_consensus.cc:359] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c9e8f4b490447caeb1e2edbc76ef6c" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36527 } }
I20260430 08:00:46.719964 17891 raft_consensus.cc:385] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:46.720059 17891 raft_consensus.cc:740] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c9e8f4b490447caeb1e2edbc76ef6c, State: Initialized, Role: FOLLOWER
I20260430 08:00:46.720427 17894 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:46.708251 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.13.94.190:36527)
0430 08:00:46.708920 (+ 669us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:46.708930 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:46.708965 (+ 35us) server_negotiation.cc:408] Connection header received
0430 08:00:46.709506 (+ 541us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:46.709522 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:46.709812 (+ 290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:46.710141 (+ 329us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:46.711080 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:46.711880 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:46.712552 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:46.712886 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:46.715553 (+ 2667us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:46.715574 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:46.715586 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:46.715616 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:46.717572 (+ 1956us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:46.718113 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:46.718118 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:46.718123 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:46.718205 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:46.718542 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:46.718545 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:46.718546 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:46.718888 (+ 342us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:46.719053 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:46.719356 (+ 303us) server_negotiation.cc:300] Negotiation successful
0430 08:00:46.719660 (+ 304us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":141,"threads_started":1}
I20260430 08:00:46.720536 17891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c9e8f4b490447caeb1e2edbc76ef6c" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36527 } }
I20260430 08:00:46.720741 17891 raft_consensus.cc:399] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:46.720829 17891 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:46.720940 17891 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:46.721874 17891 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c9e8f4b490447caeb1e2edbc76ef6c" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36527 } }
I20260430 08:00:46.722225 17891 leader_election.cc:304] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76c9e8f4b490447caeb1e2edbc76ef6c; no voters:
I20260430 08:00:46.722620 17891 leader_election.cc:290] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:46.722728 17896 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:46.722954 17896 raft_consensus.cc:697] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [term 1 LEADER]: Becoming Leader. State: Replica: 76c9e8f4b490447caeb1e2edbc76ef6c, State: Running, Role: LEADER
I20260430 08:00:46.723284 17896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c9e8f4b490447caeb1e2edbc76ef6c" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36527 } }
I20260430 08:00:46.723773 17891 sys_catalog.cc:565] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:46.724897 17898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76c9e8f4b490447caeb1e2edbc76ef6c. Latest consensus state: current_term: 1 leader_uuid: "76c9e8f4b490447caeb1e2edbc76ef6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c9e8f4b490447caeb1e2edbc76ef6c" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36527 } } }
I20260430 08:00:46.724952 17897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76c9e8f4b490447caeb1e2edbc76ef6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c9e8f4b490447caeb1e2edbc76ef6c" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36527 } } }
I20260430 08:00:46.725028 17898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:46.725050 17897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:46.725476 17905 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:46.728731 17905 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:46.735133 17905 catalog_manager.cc:1357] Generated new cluster ID: 1c9cbff21c734cac8ed54021cd6850c3
I20260430 08:00:46.735231 17905 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:46.758893 17905 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:46.760143 17905 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:46.767294 17905 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 76c9e8f4b490447caeb1e2edbc76ef6c: Generated new TSK 0
I20260430 08:00:46.768060 17905 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:46.825616 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:36527
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:46.936195 17919 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:46.936440 17919 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:46.936507 17919 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:46.940168 17919 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:46.940269 17919 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:46.940369 17919 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:46.945114 17919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:46.946218 17919 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:46.947095 17919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:46.954026 17927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:46.954097 17924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:46.954097 17925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:46.954337 17919 server_base.cc:1061] running on GCE node
I20260430 08:00:46.954854 17919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:46.955415 17919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:46.956606 17919 hybrid_clock.cc:648] HybridClock initialized: now 1777536046956579 us; error 40 us; skew 500 ppm
Apr 30 08:00:46 dist-test-slave-9bxp krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:46.959528 17919 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:00:46.960734 17919 webserver.cc:492] Webserver started at http://127.13.94.129:42181/ using document root <none> and password file <none>
I20260430 08:00:46.961344 17919 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:46.961395 17919 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:46.961615 17919 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:46.963383 17919 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "f1a7a538c2d940f0b3bdab81e4581557"
format_stamp: "Formatted at 2026-04-30 08:00:46 on dist-test-slave-9bxp"
server_key: "0e2d50b03d52286eb91f451ecae07992"
server_key_iv: "142955c26c2355d4b5b37b6ececb1ee7"
server_key_version: "encryptionkey@0"
I20260430 08:00:46.963897 17919 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "f1a7a538c2d940f0b3bdab81e4581557"
format_stamp: "Formatted at 2026-04-30 08:00:46 on dist-test-slave-9bxp"
server_key: "0e2d50b03d52286eb91f451ecae07992"
server_key_iv: "142955c26c2355d4b5b37b6ececb1ee7"
server_key_version: "encryptionkey@0"
I20260430 08:00:46.967453 17919 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260430 08:00:46.969858 17934 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:46.970911 17919 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:46.971052 17919 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "f1a7a538c2d940f0b3bdab81e4581557"
format_stamp: "Formatted at 2026-04-30 08:00:46 on dist-test-slave-9bxp"
server_key: "0e2d50b03d52286eb91f451ecae07992"
server_key_iv: "142955c26c2355d4b5b37b6ececb1ee7"
server_key_version: "encryptionkey@0"
I20260430 08:00:46.971165 17919 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:46.993738 17919 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:46.996798 17919 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:46.997027 17919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:46.997646 17919 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:46.998623 17919 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:46.998703 17919 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:46.998777 17919 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:46.998827 17919 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:47.009344 17919 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:33577
I20260430 08:00:47.009361 18047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:33577 every 8 connection(s)
I20260430 08:00:47.010403 17919 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:00:47.012210 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 17919
I20260430 08:00:47.012329 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:47.012606 13690 external_mini_cluster.cc:1468] Setting key 24077a9a1778024493356f34e0ca53b8
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
I20260430 08:00:47.025887 17894 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:47.012398 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:47463 (local address 127.13.94.190:36527)
0430 08:00:47.012652 (+ 254us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:47.012656 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:47.013683 (+ 1027us) server_negotiation.cc:408] Connection header received
0430 08:00:47.014835 (+ 1152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:47.014839 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:47.014921 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:47.015035 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:47.017263 (+ 2228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.017819 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:47.018833 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.018987 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:47.021958 (+ 2971us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:47.021983 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:47.021986 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:47.022018 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:47.023664 (+ 1646us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.024245 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.024248 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.024250 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.024299 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.024621 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.024624 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.024626 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.024857 (+ 231us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:47.024960 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:47.025568 (+ 608us) server_negotiation.cc:300] Negotiation successful
0430 08:00:47.025709 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260430 08:00:47.026664 18050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:47.012871 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:36527 (local address 127.13.94.129:47463)
0430 08:00:47.013456 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:47.013497 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:47.014614 (+ 1117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:47.015224 (+ 610us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:47.015233 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:47.015664 (+ 431us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:47.017022 (+ 1358us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:47.017046 (+ 24us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.017984 (+ 938us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:47.017991 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:47.018670 (+ 679us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:47.018680 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.018927 (+ 247us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:47.019642 (+ 715us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:47.019676 (+ 34us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:47.021736 (+ 2060us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:47.023822 (+ 2086us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:47.023829 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:47.023845 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:47.024115 (+ 270us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:47.024395 (+ 280us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:47.024398 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:47.024403 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:47.024514 (+ 111us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:47.025017 (+ 503us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:47.025026 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:47.025276 (+ 250us) client_negotiation.cc:770] Sending connection context
0430 08:00:47.025517 (+ 241us) client_negotiation.cc:241] Negotiation successful
0430 08:00:47.025793 (+ 276us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":143,"threads_started":1}
I20260430 08:00:47.027952 18048 heartbeater.cc:344] Connected to a master server at 127.13.94.190:36527
I20260430 08:00:47.028219 18048 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:47.028864 18048 heartbeater.cc:507] Master 127.13.94.190:36527 requested a full tablet report, sending...
I20260430 08:00:47.030412 17850 ts_manager.cc:194] Registered new tserver with Master: f1a7a538c2d940f0b3bdab81e4581557 (127.13.94.129:33577)
I20260430 08:00:47.031608 17850 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:47463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:47.068795 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:36527
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:47.193055 18055 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:47.193286 18055 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:47.193387 18055 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:47.197003 18055 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:47.197081 18055 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:47.197212 18055 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:47.201843 18055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:47.203079 18055 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:47.203979 18055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:47.211165 18060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:47.211136 18063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:47.211124 18061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:47.211495 18055 server_base.cc:1061] running on GCE node
I20260430 08:00:47.212067 18055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:47.212987 18055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:47.214195 18055 hybrid_clock.cc:648] HybridClock initialized: now 1777536047214163 us; error 45 us; skew 500 ppm
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536047, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:47.218084 18055 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:00:47.219383 18055 webserver.cc:492] Webserver started at http://127.13.94.130:39853/ using document root <none> and password file <none>
I20260430 08:00:47.220151 18055 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:47.220208 18055 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:47.220448 18055 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:47.222337 18055 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "c550cf3a64b94b2b952c29f6009539c8"
format_stamp: "Formatted at 2026-04-30 08:00:47 on dist-test-slave-9bxp"
server_key: "131ac87977bb609807a27e9218585973"
server_key_iv: "6791e3673bc5d9c6382bc5fa991ab73d"
server_key_version: "encryptionkey@0"
I20260430 08:00:47.222887 18055 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "c550cf3a64b94b2b952c29f6009539c8"
format_stamp: "Formatted at 2026-04-30 08:00:47 on dist-test-slave-9bxp"
server_key: "131ac87977bb609807a27e9218585973"
server_key_iv: "6791e3673bc5d9c6382bc5fa991ab73d"
server_key_version: "encryptionkey@0"
I20260430 08:00:47.227105 18055 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260430 08:00:47.229614 18070 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:47.230795 18055 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:47.230938 18055 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "c550cf3a64b94b2b952c29f6009539c8"
format_stamp: "Formatted at 2026-04-30 08:00:47 on dist-test-slave-9bxp"
server_key: "131ac87977bb609807a27e9218585973"
server_key_iv: "6791e3673bc5d9c6382bc5fa991ab73d"
server_key_version: "encryptionkey@0"
I20260430 08:00:47.231053 18055 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:47.249945 18055 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:47.253137 18055 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:47.253355 18055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:47.253959 18055 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:47.254869 18055 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:47.254941 18055 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:47.255009 18055 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:47.255075 18055 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260430 08:00:47.267400 18055 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:32879
I20260430 08:00:47.267413 18183 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:32879 every 8 connection(s)
I20260430 08:00:47.268568 18055 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:00:47.275532 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 18055
I20260430 08:00:47.275663 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:47.275899 13690 external_mini_cluster.cc:1468] Setting key 3930e2535d914ab22d8854b832727359
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536047, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:47.284739 17894 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:47.270907 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:35507 (local address 127.13.94.190:36527)
0430 08:00:47.271088 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:47.271093 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:47.271849 (+ 756us) server_negotiation.cc:408] Connection header received
0430 08:00:47.272774 (+ 925us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:47.272781 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:47.272879 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:47.272987 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:47.274588 (+ 1601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.275561 (+ 973us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:47.276684 (+ 1123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.276927 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:47.279903 (+ 2976us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:47.279923 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:47.279926 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:47.279955 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:47.282002 (+ 2047us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.282752 (+ 750us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.282758 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.282762 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.282831 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.283235 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.283241 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.283245 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.283479 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:47.283603 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:47.284330 (+ 727us) server_negotiation.cc:300] Negotiation successful
0430 08:00:47.284488 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260430 08:00:47.285939 18186 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:47.271218 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:36527 (local address 127.13.94.130:35507)
0430 08:00:47.271656 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:47.271700 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:47.272542 (+ 842us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:47.273152 (+ 610us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:47.273160 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:47.273542 (+ 382us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:47.274420 (+ 878us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:47.274433 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.275745 (+ 1312us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:47.275752 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:47.276502 (+ 750us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:47.276512 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.276836 (+ 324us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:47.277551 (+ 715us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:47.277584 (+ 33us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:47.279703 (+ 2119us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:47.282162 (+ 2459us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:47.282173 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:47.282192 (+ 19us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:47.282609 (+ 417us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:47.282949 (+ 340us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:47.282955 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:47.282960 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:47.283117 (+ 157us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:47.283612 (+ 495us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:47.283622 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:47.283977 (+ 355us) client_negotiation.cc:770] Sending connection context
0430 08:00:47.284257 (+ 280us) client_negotiation.cc:241] Negotiation successful
0430 08:00:47.284562 (+ 305us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
I20260430 08:00:47.287266 18184 heartbeater.cc:344] Connected to a master server at 127.13.94.190:36527
I20260430 08:00:47.287559 18184 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:47.288203 18184 heartbeater.cc:507] Master 127.13.94.190:36527 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
I20260430 08:00:47.289609 17850 ts_manager.cc:194] Registered new tserver with Master: c550cf3a64b94b2b952c29f6009539c8 (127.13.94.130:32879)
I20260430 08:00:47.290256 17850 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:35507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:47.336479 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:36527
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:47.443579 18191 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:47.443861 18191 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:47.443950 18191 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:47.447991 18191 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:47.448117 18191 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:47.448237 18191 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:47.452966 18191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:47.454182 18191 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:47.455085 18191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:47.462028 18196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:47.462021 18197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:47.462127 18191 server_base.cc:1061] running on GCE node
W20260430 08:00:47.462024 18199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:47.462651 18191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:47.463239 18191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:47.464401 18191 hybrid_clock.cc:648] HybridClock initialized: now 1777536047464370 us; error 47 us; skew 500 ppm
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536047, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:47.467391 18191 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:00:47.468508 18191 webserver.cc:492] Webserver started at http://127.13.94.131:46097/ using document root <none> and password file <none>
I20260430 08:00:47.469100 18191 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:47.469149 18191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:47.469306 18191 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:47.471056 18191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "d433ca7503eb4a7da7ef84796268cfee"
format_stamp: "Formatted at 2026-04-30 08:00:47 on dist-test-slave-9bxp"
server_key: "7d2f1bb9533c6abe36ce8bd54b09d887"
server_key_iv: "bf0886633a6c19fb23bcbd2266ed23c8"
server_key_version: "encryptionkey@0"
I20260430 08:00:47.471553 18191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "d433ca7503eb4a7da7ef84796268cfee"
format_stamp: "Formatted at 2026-04-30 08:00:47 on dist-test-slave-9bxp"
server_key: "7d2f1bb9533c6abe36ce8bd54b09d887"
server_key_iv: "bf0886633a6c19fb23bcbd2266ed23c8"
server_key_version: "encryptionkey@0"
I20260430 08:00:47.475153 18191 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:47.477511 18206 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:47.478554 18191 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:47.478678 18191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "d433ca7503eb4a7da7ef84796268cfee"
format_stamp: "Formatted at 2026-04-30 08:00:47 on dist-test-slave-9bxp"
server_key: "7d2f1bb9533c6abe36ce8bd54b09d887"
server_key_iv: "bf0886633a6c19fb23bcbd2266ed23c8"
server_key_version: "encryptionkey@0"
I20260430 08:00:47.478785 18191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:47.490495 18191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:47.493218 18191 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:47.493422 18191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:47.494006 18191 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:47.494879 18191 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:47.494926 18191 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:47.494966 18191 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:47.495018 18191 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:47.504786 18191 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:39145
I20260430 08:00:47.504864 18319 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:39145 every 8 connection(s)
I20260430 08:00:47.505828 18191 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:00:47.513593 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 18191
I20260430 08:00:47.513724 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:47.514015 13690 external_mini_cluster.cc:1468] Setting key 57053193791640941ce4a1ff6123f2ad
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536047, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:47.520961 17894 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:47.507890 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:53917 (local address 127.13.94.190:36527)
0430 08:00:47.508113 (+ 223us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:47.508117 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:47.509033 (+ 916us) server_negotiation.cc:408] Connection header received
0430 08:00:47.510368 (+ 1335us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:47.510390 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:47.510458 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:47.510567 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:47.512324 (+ 1757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.513017 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:47.513915 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.514303 (+ 388us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:47.516963 (+ 2660us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:47.516977 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:47.516980 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:47.517010 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:47.518903 (+ 1893us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.519426 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.519429 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.519430 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.519519 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.519898 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.519901 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.519903 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.520058 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:47.520141 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:47.520649 (+ 508us) server_negotiation.cc:300] Negotiation successful
0430 08:00:47.520832 (+ 183us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260430 08:00:47.521647 18322 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:47.508248 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:36527 (local address 127.13.94.131:53917)
0430 08:00:47.508868 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:47.508913 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:47.510107 (+ 1194us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:47.510739 (+ 632us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:47.510747 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:47.511121 (+ 374us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:47.512147 (+ 1026us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:47.512165 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.513151 (+ 986us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:47.513156 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:47.513739 (+ 583us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:47.513748 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.513999 (+ 251us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:47.514661 (+ 662us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:47.514686 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:47.516790 (+ 2104us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:47.519030 (+ 2240us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:47.519036 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:47.519046 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:47.519303 (+ 257us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:47.519656 (+ 353us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:47.519659 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:47.519661 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:47.519790 (+ 129us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:47.520154 (+ 364us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:47.520159 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:47.520393 (+ 234us) client_negotiation.cc:770] Sending connection context
0430 08:00:47.520580 (+ 187us) client_negotiation.cc:241] Negotiation successful
0430 08:00:47.520888 (+ 308us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":148,"threads_started":1}
I20260430 08:00:47.522881 18320 heartbeater.cc:344] Connected to a master server at 127.13.94.190:36527
I20260430 08:00:47.523141 18320 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:47.523684 18320 heartbeater.cc:507] Master 127.13.94.190:36527 requested a full tablet report, sending...
I20260430 08:00:47.524847 17850 ts_manager.cc:194] Registered new tserver with Master: d433ca7503eb4a7da7ef84796268cfee (127.13.94.131:39145)
I20260430 08:00:47.525390 17850 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:53917
I20260430 08:00:47.529551 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[17807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:00:47.554811 18327 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:47.545056 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.13.94.129:33577)
0430 08:00:47.545380 (+ 324us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:47.545391 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:47.545412 (+ 21us) server_negotiation.cc:408] Connection header received
0430 08:00:47.545466 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:47.545470 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:47.545642 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:47.545771 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:47.546725 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.547469 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:47.548210 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.548391 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:47.550908 (+ 2517us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:47.550942 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:47.550954 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:47.550989 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:47.553170 (+ 2181us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.553719 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.553724 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.553726 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.553778 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.553996 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.553998 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.553999 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.554182 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:47.554347 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:47.554510 (+ 163us) server_negotiation.cc:300] Negotiation successful
0430 08:00:47.554649 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":131,"threads_started":1}
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[17807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:47.566448 17894 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:47.557883 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.13.94.190:36527)
0430 08:00:47.558044 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:47.558048 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:47.558065 (+ 17us) server_negotiation.cc:408] Connection header received
0430 08:00:47.558195 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:47.558198 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:47.558244 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:47.558315 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:47.559107 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.559624 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:47.560245 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:47.560428 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:47.562738 (+ 2310us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:47.562761 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:47.562763 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:47.562802 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:47.564424 (+ 1622us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.564961 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.564972 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.564976 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.565040 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:47.565433 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:47.565455 (+ 22us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:47.565457 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:47.565727 (+ 270us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:47.565900 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:47.566138 (+ 238us) server_negotiation.cc:300] Negotiation successful
0430 08:00:47.566266 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260430 08:00:47.566974 17850 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55192
I20260430 08:00:47.567759 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 17919
I20260430 08:00:47.574332 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 18055
I20260430 08:00:47.580283 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 18191
I20260430 08:00:47.586275 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 17823
2026-04-30T08:00:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3211 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[18332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[18332](info): set up 2 sockets
Apr 30 08:00:47 dist-test-slave-9bxp krb5kdc[18332](info): commencing operation
krb5kdc: starting...
W20260430 08:00:49.605414 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.996s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:00:49 dist-test-slave-9bxp krb5kdc[18332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:00:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:49.752442 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:36487
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:36487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:49.858006 18348 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:49.858291 18348 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:49.858403 18348 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:49.861994 18348 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:49.862102 18348 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:49.862149 18348 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:49.862212 18348 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:49.862257 18348 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:49.866789 18348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:36487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:36487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:49.867972 18348 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:49.868944 18348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:49.874545 18356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:49.874658 18353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:49.874671 18354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:49.874743 18348 server_base.cc:1061] running on GCE node
I20260430 08:00:49.875268 18348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:49.876264 18348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:49.877453 18348 hybrid_clock.cc:648] HybridClock initialized: now 1777536049877426 us; error 41 us; skew 500 ppm
Apr 30 08:00:49 dist-test-slave-9bxp krb5kdc[18332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:49.880084 18348 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:00:49.880892 18348 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:49.880975 18348 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:49.881202 18348 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:49.883024 18348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9"
format_stamp: "Formatted at 2026-04-30 08:00:49 on dist-test-slave-9bxp"
server_key: "3a92474e06f519f295bd28d2d91737ca"
server_key_iv: "8507cdd5b67e510dac710cf8e41c4e4e"
server_key_version: "encryptionkey@0"
I20260430 08:00:49.883548 18348 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9"
format_stamp: "Formatted at 2026-04-30 08:00:49 on dist-test-slave-9bxp"
server_key: "3a92474e06f519f295bd28d2d91737ca"
server_key_iv: "8507cdd5b67e510dac710cf8e41c4e4e"
server_key_version: "encryptionkey@0"
I20260430 08:00:49.887243 18348 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260430 08:00:49.889722 18362 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:49.890982 18348 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:49.891157 18348 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9"
format_stamp: "Formatted at 2026-04-30 08:00:49 on dist-test-slave-9bxp"
server_key: "3a92474e06f519f295bd28d2d91737ca"
server_key_iv: "8507cdd5b67e510dac710cf8e41c4e4e"
server_key_version: "encryptionkey@0"
I20260430 08:00:49.891281 18348 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:49.909025 18348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:49.915895 18348 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:49.916169 18348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:49.924443 18348 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:36487
I20260430 08:00:49.924558 18414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:36487 every 8 connection(s)
I20260430 08:00:49.925194 18348 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:49.927937 18415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:49.928480 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 18348
I20260430 08:00:49.928640 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:49.928979 13690 external_mini_cluster.cc:1468] Setting key 10b86d642cdf33d8bf9702f8f33d1de0
I20260430 08:00:49.934027 18415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9: Bootstrap starting.
I20260430 08:00:49.936383 18415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9: Neither blocks nor log segments found. Creating new log.
Apr 30 08:00:49 dist-test-slave-9bxp krb5kdc[18332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:49.937328 18415 log.cc:826] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:49.939553 18415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9: No bootstrap required, opened a new log
I20260430 08:00:49.942288 18415 raft_consensus.cc:359] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36487 } }
I20260430 08:00:49.942478 18415 raft_consensus.cc:385] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:49.942513 18415 raft_consensus.cc:740] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d8764ed0ae43fb8676ce2fdd9e03d9, State: Initialized, Role: FOLLOWER
I20260430 08:00:49.942587 18418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:49.930583 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.13.94.190:36487)
0430 08:00:49.931051 (+ 468us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:49.931061 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:49.931096 (+ 35us) server_negotiation.cc:408] Connection header received
0430 08:00:49.931654 (+ 558us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:49.931671 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:49.931956 (+ 285us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:49.932301 (+ 345us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:49.933282 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:49.934011 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:49.934684 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:49.934996 (+ 312us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:49.937643 (+ 2647us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:49.937663 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:49.937675 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:49.937707 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:49.939584 (+ 1877us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:49.940152 (+ 568us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:49.940158 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:49.940162 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:49.940237 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:49.940506 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:49.940510 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:49.940512 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:49.940951 (+ 439us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:49.941160 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:49.941456 (+ 296us) server_negotiation.cc:300] Negotiation successful
0430 08:00:49.941719 (+ 263us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":149,"threads_started":1}
I20260430 08:00:49.942987 18415 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36487 } }
I20260430 08:00:49.943138 18415 raft_consensus.cc:399] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:49.943221 18415 raft_consensus.cc:493] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:49.943336 18415 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:49.944433 18415 raft_consensus.cc:515] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36487 } }
I20260430 08:00:49.944876 18415 leader_election.cc:304] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d8764ed0ae43fb8676ce2fdd9e03d9; no voters:
I20260430 08:00:49.945204 18415 leader_election.cc:290] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:49.945350 18420 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:49.945585 18420 raft_consensus.cc:697] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [term 1 LEADER]: Becoming Leader. State: Replica: e9d8764ed0ae43fb8676ce2fdd9e03d9, State: Running, Role: LEADER
I20260430 08:00:49.945915 18420 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36487 } }
I20260430 08:00:49.946424 18415 sys_catalog.cc:565] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:49.947481 18421 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36487 } } }
I20260430 08:00:49.947775 18421 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:49.947686 18422 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d8764ed0ae43fb8676ce2fdd9e03d9. Latest consensus state: current_term: 1 leader_uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8764ed0ae43fb8676ce2fdd9e03d9" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 36487 } } }
I20260430 08:00:49.948096 18422 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:49.948202 18429 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:49.951622 18429 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:49.957441 18429 catalog_manager.cc:1357] Generated new cluster ID: 9f8d918481324e1d998ff26c4228c319
I20260430 08:00:49.957525 18429 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:49.975373 18429 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:49.976462 18429 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:49.983063 18429 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9: Generated new TSK 0
I20260430 08:00:49.983733 18429 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:50.044937 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:36487
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:50.151597 18443 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:50.151825 18443 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:50.151885 18443 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:50.155164 18443 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:50.155243 18443 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:50.155330 18443 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:50.159721 18443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:36487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:50.160980 18443 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:50.161847 18443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:50.168298 18451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:50.168464 18448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:50.168586 18449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:50.168735 18443 server_base.cc:1061] running on GCE node
I20260430 08:00:50.169113 18443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:50.169708 18443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:50.170868 18443 hybrid_clock.cc:648] HybridClock initialized: now 1777536050170843 us; error 39 us; skew 500 ppm
Apr 30 08:00:50 dist-test-slave-9bxp krb5kdc[18332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536050, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:50.173583 18443 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:00:50.174357 18443 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:50.174415 18443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:50.174644 18443 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:50.176415 18443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "5822969b87494e269542a16dd6566d37"
format_stamp: "Formatted at 2026-04-30 08:00:50 on dist-test-slave-9bxp"
server_key: "efb34798a3ec3852cebeadcb0cb95b92"
server_key_iv: "bcbea78c41a614c2479709ce1f384bbb"
server_key_version: "encryptionkey@0"
I20260430 08:00:50.176991 18443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "5822969b87494e269542a16dd6566d37"
format_stamp: "Formatted at 2026-04-30 08:00:50 on dist-test-slave-9bxp"
server_key: "efb34798a3ec3852cebeadcb0cb95b92"
server_key_iv: "bcbea78c41a614c2479709ce1f384bbb"
server_key_version: "encryptionkey@0"
I20260430 08:00:50.180614 18443 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260430 08:00:50.183010 18457 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:50.184067 18443 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:50.184207 18443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "5822969b87494e269542a16dd6566d37"
format_stamp: "Formatted at 2026-04-30 08:00:50 on dist-test-slave-9bxp"
server_key: "efb34798a3ec3852cebeadcb0cb95b92"
server_key_iv: "bcbea78c41a614c2479709ce1f384bbb"
server_key_version: "encryptionkey@0"
I20260430 08:00:50.184326 18443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:50.203294 18443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:50.205960 18443 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:50.206182 18443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:50.206764 18443 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:50.207667 18443 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:50.207741 18443 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:50.207836 18443 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:50.207893 18443 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:50.217563 18443 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:36595
I20260430 08:00:50.217677 18570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:36595 every 8 connection(s)
I20260430 08:00:50.218206 18443 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:00:50.221274 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 18443
I20260430 08:00:50.221401 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:50.221745 13690 external_mini_cluster.cc:1468] Setting key c5996db289c61278e49487e1269371b8
Apr 30 08:00:50 dist-test-slave-9bxp krb5kdc[18332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536050, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
I20260430 08:00:50.232362 18418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.219916 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:45587 (local address 127.13.94.190:36487)
0430 08:00:50.220078 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:50.220083 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:50.220875 (+ 792us) server_negotiation.cc:408] Connection header received
0430 08:00:50.222524 (+ 1649us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:50.222529 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:50.222598 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:50.222676 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:50.224195 (+ 1519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.224743 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.225533 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.225795 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.228355 (+ 2560us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:50.228375 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:50.228378 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:50.228407 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:50.229953 (+ 1546us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:50.230667 (+ 714us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:50.230670 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:50.230672 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:50.230733 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:50.231138 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:50.231141 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:50.231142 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:50.231306 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:50.231391 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:50.232102 (+ 711us) server_negotiation.cc:300] Negotiation successful
0430 08:00:50.232228 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260430 08:00:50.233294 18573 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.220235 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:36487 (local address 127.13.94.129:45587)
0430 08:00:50.220673 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:50.220774 (+ 101us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:50.222340 (+ 1566us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:50.222882 (+ 542us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:50.222893 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:50.223354 (+ 461us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:50.224038 (+ 684us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.224056 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.224877 (+ 821us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.224885 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:50.225404 (+ 519us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.225412 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.225577 (+ 165us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.226151 (+ 574us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:50.226170 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:50.228160 (+ 1990us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:50.230101 (+ 1941us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:50.230111 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:50.230131 (+ 20us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:50.230504 (+ 373us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:50.230849 (+ 345us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:50.230853 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:50.230856 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:50.231021 (+ 165us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:50.231434 (+ 413us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:50.231441 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:50.231769 (+ 328us) client_negotiation.cc:770] Sending connection context
0430 08:00:50.232041 (+ 272us) client_negotiation.cc:241] Negotiation successful
0430 08:00:50.232285 (+ 244us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260430 08:00:50.234560 18571 heartbeater.cc:344] Connected to a master server at 127.13.94.190:36487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
I20260430 08:00:50.234907 18571 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:50.235551 18571 heartbeater.cc:507] Master 127.13.94.190:36487 requested a full tablet report, sending...
I20260430 08:00:50.237323 18379 ts_manager.cc:194] Registered new tserver with Master: 5822969b87494e269542a16dd6566d37 (127.13.94.129:36595)
I20260430 08:00:50.238459 18379 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:45587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:50.278005 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:36487
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:50.386415 18578 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:50.386670 18578 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:50.386773 18578 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:50.390342 18578 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:50.390420 18578 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:50.390563 18578 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:50.395489 18578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:36487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:50.396801 18578 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:50.397694 18578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:50.404320 18584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:50.404356 18586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:50.404476 18578 server_base.cc:1061] running on GCE node
W20260430 08:00:50.404341 18583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:50.405145 18578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:50.405901 18578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:50.407084 18578 hybrid_clock.cc:648] HybridClock initialized: now 1777536050407057 us; error 42 us; skew 500 ppm
Apr 30 08:00:50 dist-test-slave-9bxp krb5kdc[18332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536050, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:50.410283 18578 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:00:50.411118 18578 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:50.411178 18578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:50.411394 18578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:50.413718 18578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "75e98aff03ab4469a7049068cf56ec59"
format_stamp: "Formatted at 2026-04-30 08:00:50 on dist-test-slave-9bxp"
server_key: "ad3e0c378cc440e216354f774e3e9d1b"
server_key_iv: "595dfc22dd364d3cf14caa5dea55baa9"
server_key_version: "encryptionkey@0"
I20260430 08:00:50.414307 18578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "75e98aff03ab4469a7049068cf56ec59"
format_stamp: "Formatted at 2026-04-30 08:00:50 on dist-test-slave-9bxp"
server_key: "ad3e0c378cc440e216354f774e3e9d1b"
server_key_iv: "595dfc22dd364d3cf14caa5dea55baa9"
server_key_version: "encryptionkey@0"
I20260430 08:00:50.418269 18578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 08:00:50.420477 18592 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:50.421537 18578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:50.421645 18578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "75e98aff03ab4469a7049068cf56ec59"
format_stamp: "Formatted at 2026-04-30 08:00:50 on dist-test-slave-9bxp"
server_key: "ad3e0c378cc440e216354f774e3e9d1b"
server_key_iv: "595dfc22dd364d3cf14caa5dea55baa9"
server_key_version: "encryptionkey@0"
I20260430 08:00:50.421729 18578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:50.431741 18578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:50.434396 18578 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:50.434547 18578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:50.435029 18578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:50.435820 18578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:50.435868 18578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:50.435905 18578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:50.435920 18578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:50.446028 18578 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:40333
I20260430 08:00:50.446060 18705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:40333 every 8 connection(s)
I20260430 08:00:50.446640 18578 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:00:50.454524 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 18578
I20260430 08:00:50.454664 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:50.454926 13690 external_mini_cluster.cc:1468] Setting key 8714261da6ee6ac83c1f655d6414b731
Apr 30 08:00:50 dist-test-slave-9bxp krb5kdc[18332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536050, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:50.459856 18418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.448603 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:56283 (local address 127.13.94.190:36487)
0430 08:00:50.448785 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:50.448789 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:50.449503 (+ 714us) server_negotiation.cc:408] Connection header received
0430 08:00:50.450818 (+ 1315us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:50.450821 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:50.450874 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:50.450954 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:50.452228 (+ 1274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.452835 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.453510 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.453690 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.456156 (+ 2466us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:50.456199 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:50.456211 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:50.456243 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:50.457866 (+ 1623us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:50.458395 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:50.458399 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:50.458403 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:50.458454 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:50.458785 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:50.458789 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:50.458791 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:50.458949 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:50.459050 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:50.459544 (+ 494us) server_negotiation.cc:300] Negotiation successful
0430 08:00:50.459673 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 08:00:50.460521 18708 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.448947 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:36487 (local address 127.13.94.130:56283)
0430 08:00:50.449351 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:50.449385 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:50.450618 (+ 1233us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:50.451112 (+ 494us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:50.451120 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:50.451506 (+ 386us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:50.452058 (+ 552us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.452071 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.452969 (+ 898us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.452978 (+ 9us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:50.453377 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.453383 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.453628 (+ 245us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.454293 (+ 665us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:50.454314 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:50.455997 (+ 1683us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:50.458015 (+ 2018us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:50.458021 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:50.458034 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:50.458285 (+ 251us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:50.458563 (+ 278us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:50.458566 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:50.458567 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:50.458679 (+ 112us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:50.459064 (+ 385us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:50.459071 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:50.459298 (+ 227us) client_negotiation.cc:770] Sending connection context
0430 08:00:50.459512 (+ 214us) client_negotiation.cc:241] Negotiation successful
0430 08:00:50.459765 (+ 253us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":95,"threads_started":1}
I20260430 08:00:50.461732 18706 heartbeater.cc:344] Connected to a master server at 127.13.94.190:36487
I20260430 08:00:50.462005 18706 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:50.462484 18706 heartbeater.cc:507] Master 127.13.94.190:36487 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
I20260430 08:00:50.463577 18379 ts_manager.cc:194] Registered new tserver with Master: 75e98aff03ab4469a7049068cf56ec59 (127.13.94.130:40333)
I20260430 08:00:50.464237 18379 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:56283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:00:50.506206 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:36487
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:00:50.610756 18713 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:50.611040 18713 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:50.611155 18713 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:50.614674 18713 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:50.614785 18713 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:50.614907 18713 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:50.619319 18713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:36487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:50.620478 18713 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:50.621380 18713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:50.627871 18718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:50.627864 18719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:50.628005 18713 server_base.cc:1061] running on GCE node
W20260430 08:00:50.627868 18721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:50.628438 18713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:50.629050 18713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:50.630203 18713 hybrid_clock.cc:648] HybridClock initialized: now 1777536050630189 us; error 26 us; skew 500 ppm
Apr 30 08:00:50 dist-test-slave-9bxp krb5kdc[18332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536050, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:00:50.632972 18713 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:00:50.633736 18713 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:50.633793 18713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:50.634059 18713 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:50.635736 18713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "ef0793424080477ea088fec7d13e8613"
format_stamp: "Formatted at 2026-04-30 08:00:50 on dist-test-slave-9bxp"
server_key: "68b958bd20b1fa75ac62e7be376d7eb7"
server_key_iv: "73661fe8f4cab939dce20aff93f4bb5d"
server_key_version: "encryptionkey@0"
I20260430 08:00:50.636207 18713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "ef0793424080477ea088fec7d13e8613"
format_stamp: "Formatted at 2026-04-30 08:00:50 on dist-test-slave-9bxp"
server_key: "68b958bd20b1fa75ac62e7be376d7eb7"
server_key_iv: "73661fe8f4cab939dce20aff93f4bb5d"
server_key_version: "encryptionkey@0"
I20260430 08:00:50.639453 18713 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:50.641637 18727 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:50.642660 18713 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:50.642748 18713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "ef0793424080477ea088fec7d13e8613"
format_stamp: "Formatted at 2026-04-30 08:00:50 on dist-test-slave-9bxp"
server_key: "68b958bd20b1fa75ac62e7be376d7eb7"
server_key_iv: "73661fe8f4cab939dce20aff93f4bb5d"
server_key_version: "encryptionkey@0"
I20260430 08:00:50.642815 18713 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:50.656034 18713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:50.658994 18713 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:50.659176 18713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:50.659787 18713 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:50.660679 18713 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:50.660809 18713 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:50.660851 18713 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:50.660874 18713 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:50.670224 18713 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:44763
I20260430 08:00:50.670256 18840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:44763 every 8 connection(s)
I20260430 08:00:50.670835 18713 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:00:50.672142 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 18713
I20260430 08:00:50.672276 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:50.672670 13690 external_mini_cluster.cc:1468] Setting key 429372970a9bd05f8648cd941d47549d
Apr 30 08:00:50 dist-test-slave-9bxp krb5kdc[18332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536050, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:00:50.683651 18418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.673023 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:54029 (local address 127.13.94.190:36487)
0430 08:00:50.673172 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:50.673177 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:50.673865 (+ 688us) server_negotiation.cc:408] Connection header received
0430 08:00:50.675411 (+ 1546us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:50.675414 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:50.675468 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:50.675542 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:50.676861 (+ 1319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.677386 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.678063 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.678245 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.680448 (+ 2203us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:50.680466 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:50.680469 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:50.680496 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:50.681822 (+ 1326us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:50.682269 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:50.682271 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:50.682273 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:50.682316 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:50.682654 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:50.682657 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:50.682658 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:50.682802 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:50.682901 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:50.683373 (+ 472us) server_negotiation.cc:300] Negotiation successful
0430 08:00:50.683512 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260430 08:00:50.684293 18843 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.673249 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:36487 (local address 127.13.94.131:54029)
0430 08:00:50.673737 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:50.673780 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:50.675225 (+ 1445us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:50.675820 (+ 595us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:50.675829 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:50.676207 (+ 378us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:50.676667 (+ 460us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.676678 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.677517 (+ 839us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.677521 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:50.677941 (+ 420us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.677947 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.678166 (+ 219us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.678761 (+ 595us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:00:50.678780 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:50.680292 (+ 1512us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:50.681938 (+ 1646us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:50.681943 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:50.681953 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:50.682180 (+ 227us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:50.682411 (+ 231us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:00:50.682413 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:00:50.682415 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:00:50.682514 (+ 99us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:00:50.682929 (+ 415us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:50.682934 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:50.683150 (+ 216us) client_negotiation.cc:770] Sending connection context
0430 08:00:50.683335 (+ 185us) client_negotiation.cc:241] Negotiation successful
0430 08:00:50.683525 (+ 190us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
I20260430 08:00:50.685492 18841 heartbeater.cc:344] Connected to a master server at 127.13.94.190:36487
I20260430 08:00:50.685823 18841 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:50.686331 18841 heartbeater.cc:507] Master 127.13.94.190:36487 requested a full tablet report, sending...
I20260430 08:00:50.687353 18379 ts_manager.cc:194] Registered new tserver with Master: ef0793424080477ea088fec7d13e8613 (127.13.94.131:44763)
I20260430 08:00:50.687909 18379 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:54029
I20260430 08:00:50.698799 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 08:00:50.708297 18418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.700865 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.13.94.190:36487)
0430 08:00:50.701060 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:50.701065 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:50.701309 (+ 244us) server_negotiation.cc:408] Connection header received
0430 08:00:50.701489 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:50.701493 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:50.701544 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:50.701640 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:50.702554 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.703080 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.703797 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.704016 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.705206 (+ 1190us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:50.705230 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:50.705233 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:00:50.705259 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:50.706589 (+ 1330us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:50.707046 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:50.707051 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:50.707054 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:50.707112 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:00:50.707384 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:00:50.707388 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:00:50.707390 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:00:50.707583 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:50.707734 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:50.707978 (+ 244us) server_negotiation.cc:300] Negotiation successful
0430 08:00:50.708103 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260430 08:00:50.711174 18379 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:00:50.713730 18379 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:50.727247 18852 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.722546 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:44763 (local address 127.0.0.1:33432)
0430 08:00:50.723016 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:50.723059 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:50.723213 (+ 154us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:50.724180 (+ 967us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:50.724185 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:50.724209 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:50.724551 (+ 342us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.724559 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.725823 (+ 1264us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.725827 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:50.726724 (+ 897us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.726733 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.726840 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.726906 (+ 66us) client_negotiation.cc:770] Sending connection context
0430 08:00:50.726988 (+ 82us) client_negotiation.cc:241] Negotiation successful
0430 08:00:50.727051 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"spinlock_wait_cycles":13568,"thread_start_us":102,"threads_started":1}
I20260430 08:00:50.727478 18854 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.722729 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:40333 (local address 127.0.0.1:46856)
0430 08:00:50.723304 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:50.723319 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:50.723394 (+ 75us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:50.724204 (+ 810us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:50.724207 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:50.724280 (+ 73us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:50.724609 (+ 329us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.724639 (+ 30us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.726089 (+ 1450us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.726093 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:50.727028 (+ 935us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.727036 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.727153 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.727168 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 08:00:50.727214 (+ 46us) client_negotiation.cc:241] Negotiation successful
0430 08:00:50.727263 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"spinlock_wait_cycles":85120,"thread_start_us":109,"threads_started":1}
I20260430 08:00:50.727766 18855 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.723314 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:36595 (local address 127.0.0.1:45352)
0430 08:00:50.724146 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:50.724161 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:50.724280 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:50.724901 (+ 621us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:50.724904 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:50.724925 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:50.725158 (+ 233us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.725164 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.726530 (+ 1366us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.726534 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:50.727398 (+ 864us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.727406 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.727507 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.727524 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:00:50.727591 (+ 67us) client_negotiation.cc:241] Negotiation successful
0430 08:00:50.727649 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"spinlock_wait_cycles":5760,"thread_start_us":96,"threads_started":1}
I20260430 08:00:50.728241 18856 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.723382 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.13.94.130:40333)
0430 08:00:50.723743 (+ 361us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:50.723749 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:50.723770 (+ 21us) server_negotiation.cc:408] Connection header received
0430 08:00:50.723869 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:50.723873 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:50.724042 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:50.724457 (+ 415us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:50.724777 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.725955 (+ 1178us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.727189 (+ 1234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.727739 (+ 550us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.728004 (+ 265us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:50.728061 (+ 57us) server_negotiation.cc:300] Negotiation successful
0430 08:00:50.728132 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":86,"threads_started":1}
I20260430 08:00:50.728950 18857 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.723569 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.13.94.129:36595)
0430 08:00:50.723932 (+ 363us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:50.723937 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:50.724361 (+ 424us) server_negotiation.cc:408] Connection header received
0430 08:00:50.724448 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:50.724452 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:50.724595 (+ 143us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:50.724749 (+ 154us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:50.725295 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.726399 (+ 1104us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.727962 (+ 1563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.728502 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.728616 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:50.728682 (+ 66us) server_negotiation.cc:300] Negotiation successful
0430 08:00:50.728807 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260430 08:00:50.729023 18853 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.722656 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.13.94.131:44763)
0430 08:00:50.723667 (+ 1011us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:50.723677 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:50.723703 (+ 26us) server_negotiation.cc:408] Connection header received
0430 08:00:50.723802 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:50.723807 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:50.723999 (+ 192us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:50.724182 (+ 183us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:50.724685 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.725714 (+ 1029us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.727579 (+ 1865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.728617 (+ 1038us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.728741 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:50.728801 (+ 60us) server_negotiation.cc:300] Negotiation successful
0430 08:00:50.728881 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":106,"threads_started":1}
I20260430 08:00:50.731252 18775 tablet_service.cc:1511] Processing CreateTablet for tablet bda386415f0b4caca8d197c552ef9b6d (DEFAULT_TABLE table=test-table [id=d8edb78d0e124ea88ce23006d8f326cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:50.731230 18640 tablet_service.cc:1511] Processing CreateTablet for tablet bda386415f0b4caca8d197c552ef9b6d (DEFAULT_TABLE table=test-table [id=d8edb78d0e124ea88ce23006d8f326cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:50.732103 18775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda386415f0b4caca8d197c552ef9b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:50.732105 18640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda386415f0b4caca8d197c552ef9b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:50.735281 18505 tablet_service.cc:1511] Processing CreateTablet for tablet bda386415f0b4caca8d197c552ef9b6d (DEFAULT_TABLE table=test-table [id=d8edb78d0e124ea88ce23006d8f326cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:50.736294 18505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda386415f0b4caca8d197c552ef9b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:50.737742 18858 tablet_bootstrap.cc:492] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: Bootstrap starting.
I20260430 08:00:50.739018 18859 tablet_bootstrap.cc:492] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: Bootstrap starting.
I20260430 08:00:50.740562 18858 tablet_bootstrap.cc:654] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:50.741098 18859 tablet_bootstrap.cc:654] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:50.741454 18858 log.cc:826] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:50.741751 18860 tablet_bootstrap.cc:492] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37: Bootstrap starting.
I20260430 08:00:50.741873 18859 log.cc:826] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:50.743614 18860 tablet_bootstrap.cc:654] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:50.743726 18858 tablet_bootstrap.cc:492] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: No bootstrap required, opened a new log
I20260430 08:00:50.743726 18859 tablet_bootstrap.cc:492] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: No bootstrap required, opened a new log
I20260430 08:00:50.743919 18858 ts_tablet_manager.cc:1403] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260430 08:00:50.743919 18859 ts_tablet_manager.cc:1403] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260430 08:00:50.744412 18860 log.cc:826] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:50.746037 18860 tablet_bootstrap.cc:492] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37: No bootstrap required, opened a new log
I20260430 08:00:50.746235 18860 ts_tablet_manager.cc:1403] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260430 08:00:50.747671 18858 raft_consensus.cc:359] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } }
I20260430 08:00:50.747663 18859 raft_consensus.cc:359] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } }
I20260430 08:00:50.747887 18858 raft_consensus.cc:385] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:50.747913 18859 raft_consensus.cc:385] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:50.747982 18858 raft_consensus.cc:740] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e98aff03ab4469a7049068cf56ec59, State: Initialized, Role: FOLLOWER
I20260430 08:00:50.748045 18859 raft_consensus.cc:740] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0793424080477ea088fec7d13e8613, State: Initialized, Role: FOLLOWER
I20260430 08:00:50.748449 18859 consensus_queue.cc:260] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } }
I20260430 08:00:50.748449 18858 consensus_queue.cc:260] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } }
I20260430 08:00:50.749161 18860 raft_consensus.cc:359] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } }
I20260430 08:00:50.749353 18858 ts_tablet_manager.cc:1434] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260430 08:00:50.749353 18859 ts_tablet_manager.cc:1434] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260430 08:00:50.749385 18860 raft_consensus.cc:385] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:50.749424 18706 heartbeater.cc:499] Master 127.13.94.190:36487 was elected leader, sending a full tablet report...
I20260430 08:00:50.749464 18860 raft_consensus.cc:740] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5822969b87494e269542a16dd6566d37, State: Initialized, Role: FOLLOWER
I20260430 08:00:50.749939 18860 consensus_queue.cc:260] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } }
I20260430 08:00:50.750450 18841 heartbeater.cc:499] Master 127.13.94.190:36487 was elected leader, sending a full tablet report...
I20260430 08:00:50.750550 18571 heartbeater.cc:499] Master 127.13.94.190:36487 was elected leader, sending a full tablet report...
I20260430 08:00:50.750765 18860 ts_tablet_manager.cc:1434] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260430 08:00:50.921633 18842 tablet.cc:2404] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:00:50.947759 18707 tablet.cc:2404] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:00:50.961069 18864 raft_consensus.cc:493] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:50.961247 18864 raft_consensus.cc:515] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } }
I20260430 08:00:50.962500 18864 leader_election.cc:290] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0793424080477ea088fec7d13e8613 (127.13.94.131:44763), 5822969b87494e269542a16dd6566d37 (127.13.94.129:36595)
I20260430 08:00:50.966053 18867 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.962768 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:36595 (local address 127.13.94.130:43689)
0430 08:00:50.963112 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:50.963126 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:50.963242 (+ 116us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:50.963543 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:50.963546 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:50.963564 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:50.963806 (+ 242us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.963812 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.964964 (+ 1152us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.964970 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:50.965568 (+ 598us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.965575 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.965735 (+ 160us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.965753 (+ 18us) client_negotiation.cc:770] Sending connection context
0430 08:00:50.965809 (+ 56us) client_negotiation.cc:241] Negotiation successful
0430 08:00:50.965874 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":109,"threads_started":1}
I20260430 08:00:50.966138 18868 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.962810 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:44763 (local address 127.13.94.130:58861)
0430 08:00:50.963162 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:50.963176 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:50.963278 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:50.963655 (+ 377us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:50.963659 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:50.963685 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:00:50.963952 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.963959 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.964951 (+ 992us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.964957 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:50.965650 (+ 693us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:50.965658 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.965808 (+ 150us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.965834 (+ 26us) client_negotiation.cc:770] Sending connection context
0430 08:00:50.965899 (+ 65us) client_negotiation.cc:241] Negotiation successful
0430 08:00:50.965956 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":91,"threads_started":1}
I20260430 08:00:50.966475 18857 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.962870 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:43689 (local address 127.13.94.129:36595)
0430 08:00:50.963022 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:50.963027 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:50.963194 (+ 167us) server_negotiation.cc:408] Connection header received
0430 08:00:50.963374 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:50.963382 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:50.963429 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:50.963515 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:50.963931 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.964832 (+ 901us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.965700 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.966203 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.966239 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:50.966290 (+ 51us) server_negotiation.cc:300] Negotiation successful
0430 08:00:50.966343 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260430 08:00:50.966652 18853 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:50.962972 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:58861 (local address 127.13.94.131:44763)
0430 08:00:50.963140 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:50.963147 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:50.963234 (+ 87us) server_negotiation.cc:408] Connection header received
0430 08:00:50.963416 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:50.963429 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:50.963481 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:50.963566 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:00:50.964093 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.964797 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:50.965796 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:50.966328 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:50.966393 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:50.966458 (+ 65us) server_negotiation.cc:300] Negotiation successful
0430 08:00:50.966518 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260430 08:00:50.967139 18525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda386415f0b4caca8d197c552ef9b6d" candidate_uuid: "75e98aff03ab4469a7049068cf56ec59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5822969b87494e269542a16dd6566d37" is_pre_election: true
I20260430 08:00:50.967271 18795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda386415f0b4caca8d197c552ef9b6d" candidate_uuid: "75e98aff03ab4469a7049068cf56ec59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0793424080477ea088fec7d13e8613" is_pre_election: true
I20260430 08:00:50.967487 18795 raft_consensus.cc:2468] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e98aff03ab4469a7049068cf56ec59 in term 0.
I20260430 08:00:50.967480 18525 raft_consensus.cc:2468] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75e98aff03ab4469a7049068cf56ec59 in term 0.
I20260430 08:00:50.968050 18595 leader_election.cc:304] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5822969b87494e269542a16dd6566d37, 75e98aff03ab4469a7049068cf56ec59; no voters:
I20260430 08:00:50.968348 18864 raft_consensus.cc:2804] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:50.968433 18864 raft_consensus.cc:493] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:50.968467 18864 raft_consensus.cc:3060] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 0 FOLLOWER]: Advancing to term 1
W20260430 08:00:50.969422 18572 tablet.cc:2404] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:00:50.969669 18864 raft_consensus.cc:515] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } }
I20260430 08:00:50.970069 18864 leader_election.cc:290] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [CANDIDATE]: Term 1 election: Requested vote from peers ef0793424080477ea088fec7d13e8613 (127.13.94.131:44763), 5822969b87494e269542a16dd6566d37 (127.13.94.129:36595)
I20260430 08:00:50.970455 18525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda386415f0b4caca8d197c552ef9b6d" candidate_uuid: "75e98aff03ab4469a7049068cf56ec59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5822969b87494e269542a16dd6566d37"
I20260430 08:00:50.970455 18795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bda386415f0b4caca8d197c552ef9b6d" candidate_uuid: "75e98aff03ab4469a7049068cf56ec59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0793424080477ea088fec7d13e8613"
I20260430 08:00:50.970583 18525 raft_consensus.cc:3060] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:50.970583 18795 raft_consensus.cc:3060] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:50.971807 18525 raft_consensus.cc:2468] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e98aff03ab4469a7049068cf56ec59 in term 1.
I20260430 08:00:50.971807 18795 raft_consensus.cc:2468] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75e98aff03ab4469a7049068cf56ec59 in term 1.
I20260430 08:00:50.972194 18596 leader_election.cc:304] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75e98aff03ab4469a7049068cf56ec59, ef0793424080477ea088fec7d13e8613; no voters:
I20260430 08:00:50.972389 18864 raft_consensus.cc:2804] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:50.972594 18864 raft_consensus.cc:697] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 1 LEADER]: Becoming Leader. State: Replica: 75e98aff03ab4469a7049068cf56ec59, State: Running, Role: LEADER
I20260430 08:00:50.972944 18864 consensus_queue.cc:237] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } }
I20260430 08:00:50.976176 18379 catalog_manager.cc:5671] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75e98aff03ab4469a7049068cf56ec59 (127.13.94.130). New cstate: current_term: 1 leader_uuid: "75e98aff03ab4469a7049068cf56ec59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75e98aff03ab4469a7049068cf56ec59" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 40333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 } health_report { overall_health: UNKNOWN } } }
I20260430 08:00:51.022219 18856 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.019116 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.13.94.130:40333)
0430 08:00:51.019264 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.019267 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.019283 (+ 16us) server_negotiation.cc:408] Connection header received
0430 08:00:51.019401 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.019404 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.019462 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.019555 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:00:51.019967 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:51.020503 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:51.021219 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:51.021366 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:51.021448 (+ 82us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:00:51.021860 (+ 412us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:00:51.021952 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.022064 (+ 112us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.022107 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 08:00:51.028249 18525 raft_consensus.cc:1275] T bda386415f0b4caca8d197c552ef9b6d P 5822969b87494e269542a16dd6566d37 [term 1 FOLLOWER]: Refusing update from remote peer 75e98aff03ab4469a7049068cf56ec59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:51.028249 18795 raft_consensus.cc:1275] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [term 1 FOLLOWER]: Refusing update from remote peer 75e98aff03ab4469a7049068cf56ec59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:51.029003 18864 consensus_queue.cc:1048] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5822969b87494e269542a16dd6566d37" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:51.029246 18869 consensus_queue.cc:1048] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0793424080477ea088fec7d13e8613" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:51.036145 18872 mvcc.cc:204] Tried to move back new op lower bound from 7280787665004085248 to 7280787664787939328. Current Snapshot: MvccSnapshot[applied={T|T < 7280787665004085248}]
I20260430 08:00:51.036799 18873 mvcc.cc:204] Tried to move back new op lower bound from 7280787665004085248 to 7280787664787939328. Current Snapshot: MvccSnapshot[applied={T|T < 7280787665004085248}]
I20260430 08:00:51.045096 18378 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:51.045264 18378 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:51.047981 18378 catalog_manager.cc:5958] T 00000000000000000000000000000000 P e9d8764ed0ae43fb8676ce2fdd9e03d9: Sending DeleteTablet for 3 replicas of tablet bda386415f0b4caca8d197c552ef9b6d
I20260430 08:00:51.048899 18775 tablet_service.cc:1558] Processing DeleteTablet for tablet bda386415f0b4caca8d197c552ef9b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:51 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:33432
I20260430 08:00:51.048982 18505 tablet_service.cc:1558] Processing DeleteTablet for tablet bda386415f0b4caca8d197c552ef9b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:51 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:45352
I20260430 08:00:51.049014 18640 tablet_service.cc:1558] Processing DeleteTablet for tablet bda386415f0b4caca8d197c552ef9b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:51 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:46856
I20260430 08:00:51.049536 18883 tablet_replica.cc:333] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: stopping tablet replica
I20260430 08:00:51.049716 18882 tablet_replica.cc:333] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: stopping tablet replica
I20260430 08:00:51.049875 18883 raft_consensus.cc:2243] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:00:51.050181 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 18443
I20260430 08:00:51.050078 18882 raft_consensus.cc:2243] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:51.050316 18883 raft_consensus.cc:2272] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:51.051469 18882 raft_consensus.cc:2272] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:51.052731 18883 ts_tablet_manager.cc:1916] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:51.054446 18882 ts_tablet_manager.cc:1916] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:51.055486 18883 ts_tablet_manager.cc:1929] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:00:51.055683 18883 log.cc:1199] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/bda386415f0b4caca8d197c552ef9b6d
I20260430 08:00:51.056104 18883 ts_tablet_manager.cc:1950] T bda386415f0b4caca8d197c552ef9b6d P 75e98aff03ab4469a7049068cf56ec59: Deleting consensus metadata
I20260430 08:00:51.057541 18364 catalog_manager.cc:5002] TS 75e98aff03ab4469a7049068cf56ec59 (127.13.94.130:40333): tablet bda386415f0b4caca8d197c552ef9b6d (table test-table [id=d8edb78d0e124ea88ce23006d8f326cb]) successfully deleted
I20260430 08:00:51.057662 18882 ts_tablet_manager.cc:1929] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:00:51.057749 18882 log.cc:1199] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/bda386415f0b4caca8d197c552ef9b6d
I20260430 08:00:51.058145 18882 ts_tablet_manager.cc:1950] T bda386415f0b4caca8d197c552ef9b6d P ef0793424080477ea088fec7d13e8613: Deleting consensus metadata
W20260430 08:00:51.059230 18365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.94.129:36595 (error 108)
I20260430 08:00:51.059383 18366 catalog_manager.cc:5002] TS ef0793424080477ea088fec7d13e8613 (127.13.94.131:44763): tablet bda386415f0b4caca8d197c552ef9b6d (table test-table [id=d8edb78d0e124ea88ce23006d8f326cb]) successfully deleted
I20260430 08:00:51.060122 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 18578
I20260430 08:00:51.060411 18852 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.060091 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:36595 (local address 127.0.0.1:45368)
0430 08:00:51.060236 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:51.060298 (+ 62us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.129:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260430 08:00:51.061894 18365 catalog_manager.cc:4729] TS 5822969b87494e269542a16dd6566d37 (127.13.94.129:36595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bda386415f0b4caca8d197c552ef9b6d: Network error: Client connection negotiation failed: client connection to 127.13.94.129:36595: connect: Connection refused (error 111)
I20260430 08:00:51.066844 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 18713
I20260430 08:00:51.073462 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 18348
2026-04-30T08:00:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3488 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-04-30T08:00:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:51Z Disabled control of system clock
I20260430 08:00:51.104517 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:45303
--webserver_interface=127.13.94.190
--webserver_port=0
--builtin_ntp_servers=127.13.94.148:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:45303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260430 08:00:51.207137 18888 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:51.207376 18888 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:51.207425 18888 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:51.210882 18888 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:51.210958 18888 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:51.210973 18888 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:51.210994 18888 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:51.211011 18888 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:00:51.215016 18888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:45303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:45303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:51.216104 18888 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:51.217418 18888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:51.222980 18896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:51.223042 18893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:51.223063 18894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:51.223418 18888 server_base.cc:1061] running on GCE node
I20260430 08:00:51.223927 18888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:51.224876 18888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:51.226034 18888 hybrid_clock.cc:648] HybridClock initialized: now 1777536051226019 us; error 30 us; skew 500 ppm
I20260430 08:00:51.227854 18888 webserver.cc:492] Webserver started at http://127.13.94.190:36529/ using document root <none> and password file <none>
I20260430 08:00:51.228433 18888 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:51.228488 18888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:51.228790 18888 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:51.230513 18888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "e2a6f402da36486eaec652293468b548"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "7633b364772cb3098fea24833b337857"
server_key_iv: "aef49fafd9700eaed642d70f19fe1c60"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.231019 18888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "e2a6f402da36486eaec652293468b548"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "7633b364772cb3098fea24833b337857"
server_key_iv: "aef49fafd9700eaed642d70f19fe1c60"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.234338 18888 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:51.236585 18902 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.237583 18888 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:51.237727 18888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "e2a6f402da36486eaec652293468b548"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "7633b364772cb3098fea24833b337857"
server_key_iv: "aef49fafd9700eaed642d70f19fe1c60"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.237830 18888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:51.242476 18888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:51.243067 18888 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:51.243259 18888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:51.250540 18888 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:45303
I20260430 08:00:51.250595 18954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:45303 every 8 connection(s)
I20260430 08:00:51.251621 18888 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:51.254630 18955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:51.259438 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 18888
I20260430 08:00:51.259569 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:51.259814 13690 external_mini_cluster.cc:1468] Setting key 5c19994e5d069923a5c00ea91119527d
I20260430 08:00:51.260360 18955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548: Bootstrap starting.
I20260430 08:00:51.262837 18955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:51.263772 18955 log.cc:826] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:51.265276 18958 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.261212 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.13.94.190:45303)
0430 08:00:51.261803 (+ 591us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.261813 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.261850 (+ 37us) server_negotiation.cc:408] Connection header received
0430 08:00:51.262406 (+ 556us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.262422 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.262755 (+ 333us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.263118 (+ 363us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.264000 (+ 882us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.264005 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.264016 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.264026 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.264063 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.264073 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.264215 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.264348 (+ 133us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.264486 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":145,"threads_started":1}
I20260430 08:00:51.265978 18955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548: No bootstrap required, opened a new log
I20260430 08:00:51.269083 18955 raft_consensus.cc:359] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a6f402da36486eaec652293468b548" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45303 } }
I20260430 08:00:51.269320 18955 raft_consensus.cc:385] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:51.269413 18955 raft_consensus.cc:740] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a6f402da36486eaec652293468b548, State: Initialized, Role: FOLLOWER
I20260430 08:00:51.269862 18955 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a6f402da36486eaec652293468b548" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45303 } }
I20260430 08:00:51.270021 18955 raft_consensus.cc:399] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:51.270104 18955 raft_consensus.cc:493] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:51.270222 18955 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:51.271806 18955 raft_consensus.cc:515] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a6f402da36486eaec652293468b548" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45303 } }
I20260430 08:00:51.272190 18955 leader_election.cc:304] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2a6f402da36486eaec652293468b548; no voters:
I20260430 08:00:51.272475 18955 leader_election.cc:290] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:51.272680 18960 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:51.272948 18960 raft_consensus.cc:697] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [term 1 LEADER]: Becoming Leader. State: Replica: e2a6f402da36486eaec652293468b548, State: Running, Role: LEADER
I20260430 08:00:51.273267 18960 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a6f402da36486eaec652293468b548" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45303 } }
I20260430 08:00:51.273823 18955 sys_catalog.cc:565] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:51.274973 18962 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a6f402da36486eaec652293468b548. Latest consensus state: current_term: 1 leader_uuid: "e2a6f402da36486eaec652293468b548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a6f402da36486eaec652293468b548" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45303 } } }
I20260430 08:00:51.275115 18962 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:51.275512 18961 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2a6f402da36486eaec652293468b548" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a6f402da36486eaec652293468b548" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45303 } } }
I20260430 08:00:51.275619 18961 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548 [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:51.275496 18969 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:51.279142 18969 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:51.284848 18969 catalog_manager.cc:1357] Generated new cluster ID: 911708c3064a448595c7f708d380d115
I20260430 08:00:51.284936 18969 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:51.294008 18969 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:51.294365 18969 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:51.307687 18969 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548: Generated new TSK 0
I20260430 08:00:51.308460 18969 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 08:00:51.313994 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:45303
--builtin_ntp_servers=127.13.94.148:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260430 08:00:51.419566 18979 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:51.419834 18979 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:51.419934 18979 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:51.423813 18979 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:51.423906 18979 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:51.424014 18979 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:00:51.428007 18979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:45303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:51.429267 18979 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:51.430374 18979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:51.436807 18984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:51.436796 18987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:51.436796 18985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:51.437182 18979 server_base.cc:1061] running on GCE node
I20260430 08:00:51.437518 18979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:51.438032 18979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:51.439215 18979 hybrid_clock.cc:648] HybridClock initialized: now 1777536051439186 us; error 45 us; skew 500 ppm
I20260430 08:00:51.441035 18979 webserver.cc:492] Webserver started at http://127.13.94.129:36221/ using document root <none> and password file <none>
I20260430 08:00:51.441604 18979 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:51.441691 18979 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:51.441923 18979 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:51.443697 18979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "3a21e136f80b43d684283a702e4ba314"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "1fe74413ade2c6ac3497dc4365800690"
server_key_iv: "f315e7b705363de38d7da1a196bd2cb3"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.444206 18979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "3a21e136f80b43d684283a702e4ba314"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "1fe74413ade2c6ac3497dc4365800690"
server_key_iv: "f315e7b705363de38d7da1a196bd2cb3"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.447599 18979 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260430 08:00:51.449846 18993 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.450805 18979 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:00:51.450954 18979 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "3a21e136f80b43d684283a702e4ba314"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "1fe74413ade2c6ac3497dc4365800690"
server_key_iv: "f315e7b705363de38d7da1a196bd2cb3"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.451057 18979 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:51.455590 18979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:51.456120 18979 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:51.456296 18979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:51.456859 18979 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:51.457777 18979 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:51.457850 18979 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.457944 18979 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:51.457995 18979 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.466602 18979 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:33513
I20260430 08:00:51.466672 19106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:33513 every 8 connection(s)
I20260430 08:00:51.467607 18979 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:00:51.469997 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 18979
I20260430 08:00:51.470125 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:00:51.470430 13690 external_mini_cluster.cc:1468] Setting key 35cd6e3987c8ec861ebdf6694faa2cba
I20260430 08:00:51.472630 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:45303
--builtin_ntp_servers=127.13.94.148:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260430 08:00:51.473688 18958 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.469493 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:40467 (local address 127.13.94.190:45303)
0430 08:00:51.469690 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.469695 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.470810 (+ 1115us) server_negotiation.cc:408] Connection header received
0430 08:00:51.471803 (+ 993us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.471807 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.471849 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.471962 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.473233 (+ 1271us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.473236 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.473238 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.473248 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.473267 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.473274 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.473350 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.473531 (+ 181us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.473561 (+ 30us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260430 08:00:51.474464 19109 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.469850 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45303 (local address 127.13.94.129:40467)
0430 08:00:51.470668 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:51.470703 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:51.471521 (+ 818us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:51.472141 (+ 620us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:51.472153 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:51.472236 (+ 83us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:51.472964 (+ 728us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:00:51.472968 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:51.473003 (+ 35us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:00:51.473071 (+ 68us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:51.473365 (+ 294us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:51.473375 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:51.473383 (+ 8us) client_negotiation.cc:770] Sending connection context
0430 08:00:51.473578 (+ 195us) client_negotiation.cc:241] Negotiation successful
0430 08:00:51.473702 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":125,"threads_started":1}
I20260430 08:00:51.475900 19107 heartbeater.cc:344] Connected to a master server at 127.13.94.190:45303
I20260430 08:00:51.476146 19107 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:51.476563 19107 heartbeater.cc:507] Master 127.13.94.190:45303 requested a full tablet report, sending...
I20260430 08:00:51.478336 18919 ts_manager.cc:194] Registered new tserver with Master: 3a21e136f80b43d684283a702e4ba314 (127.13.94.129:33513)
W20260430 08:00:51.582608 19110 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:51.582896 19110 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:51.582979 19110 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:51.587356 19110 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:51.587450 19110 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:51.587560 19110 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:00:51.591617 19110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:45303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:51.592852 19110 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:51.593976 19110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:51.600884 19118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:51.600953 19110 server_base.cc:1061] running on GCE node
W20260430 08:00:51.600869 19116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:51.600869 19115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:51.601495 19110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:51.602074 19110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:51.603256 19110 hybrid_clock.cc:648] HybridClock initialized: now 1777536051603226 us; error 39 us; skew 500 ppm
I20260430 08:00:51.605448 19110 webserver.cc:492] Webserver started at http://127.13.94.130:42261/ using document root <none> and password file <none>
I20260430 08:00:51.606069 19110 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:51.606127 19110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:51.606290 19110 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:51.607986 19110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "4cdebc5dedf5494d87745bf4cc206975"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "b2aa7a4238d9ba10d35efed554d3a705"
server_key_iv: "ab07c09bc3afa6e44d51ef723bb44e83"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.608498 19110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "4cdebc5dedf5494d87745bf4cc206975"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "b2aa7a4238d9ba10d35efed554d3a705"
server_key_iv: "ab07c09bc3afa6e44d51ef723bb44e83"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.612159 19110 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260430 08:00:51.614688 19124 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.615731 19110 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:00:51.615887 19110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "4cdebc5dedf5494d87745bf4cc206975"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "b2aa7a4238d9ba10d35efed554d3a705"
server_key_iv: "ab07c09bc3afa6e44d51ef723bb44e83"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.616012 19110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:51.621239 19110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:51.621919 19110 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:51.622143 19110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:51.622736 19110 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:51.623782 19110 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:51.623858 19110 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.623929 19110 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:51.623980 19110 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.633508 19110 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:37871
I20260430 08:00:51.633525 19237 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:37871 every 8 connection(s)
I20260430 08:00:51.634675 19110 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:00:51.638965 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 19110
I20260430 08:00:51.639094 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:00:51.639376 13690 external_mini_cluster.cc:1468] Setting key 9880506812f3903af974d4ff7ef98d2f
I20260430 08:00:51.640905 18958 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.636764 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:60803 (local address 127.13.94.190:45303)
0430 08:00:51.636906 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.636909 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.637819 (+ 910us) server_negotiation.cc:408] Connection header received
0430 08:00:51.638885 (+ 1066us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.638890 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.638988 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.639117 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.640340 (+ 1223us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.640344 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.640346 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.640356 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.640378 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.640385 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.640473 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.640648 (+ 175us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.640683 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260430 08:00:51.641572 19240 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.637051 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45303 (local address 127.13.94.130:60803)
0430 08:00:51.637656 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:51.637690 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:51.638545 (+ 855us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:51.639372 (+ 827us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:51.639381 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:51.639450 (+ 69us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:51.640082 (+ 632us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:00:51.640084 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:51.640108 (+ 24us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:00:51.640187 (+ 79us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:51.640485 (+ 298us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:51.640492 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:51.640498 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 08:00:51.640753 (+ 255us) client_negotiation.cc:241] Negotiation successful
0430 08:00:51.640902 (+ 149us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":103,"threads_started":1}
I20260430 08:00:51.641768 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:45303
--builtin_ntp_servers=127.13.94.148:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260430 08:00:51.642892 19238 heartbeater.cc:344] Connected to a master server at 127.13.94.190:45303
I20260430 08:00:51.643141 19238 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:51.643555 19238 heartbeater.cc:507] Master 127.13.94.190:45303 requested a full tablet report, sending...
I20260430 08:00:51.644589 18919 ts_manager.cc:194] Registered new tserver with Master: 4cdebc5dedf5494d87745bf4cc206975 (127.13.94.130:37871)
W20260430 08:00:51.747619 19241 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:51.747869 19241 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:51.747951 19241 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:51.751439 19241 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:51.751498 19241 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:51.751612 19241 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:00:51.755694 19241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:45303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:51.756903 19241 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:51.758054 19241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:51.764767 19247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:51.764971 19241 server_base.cc:1061] running on GCE node
W20260430 08:00:51.764999 19246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:51.764873 19249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:51.765375 19241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:51.765933 19241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:51.767132 19241 hybrid_clock.cc:648] HybridClock initialized: now 1777536051767099 us; error 52 us; skew 500 ppm
I20260430 08:00:51.768944 19241 webserver.cc:492] Webserver started at http://127.13.94.131:39625/ using document root <none> and password file <none>
I20260430 08:00:51.769507 19241 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:51.769595 19241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:51.769819 19241 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:51.771451 19241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "30455943b6b54ad6a00f051f9aac83fc"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "6a9235b49ca35aba0976c721f9aa057d"
server_key_iv: "59c62078fd1575d535bc2fa88a71c46a"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.771926 19241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "30455943b6b54ad6a00f051f9aac83fc"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "6a9235b49ca35aba0976c721f9aa057d"
server_key_iv: "59c62078fd1575d535bc2fa88a71c46a"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.775292 19241 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260430 08:00:51.777704 19255 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.778817 19241 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:51.778932 19241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "30455943b6b54ad6a00f051f9aac83fc"
format_stamp: "Formatted at 2026-04-30 08:00:51 on dist-test-slave-9bxp"
server_key: "6a9235b49ca35aba0976c721f9aa057d"
server_key_iv: "59c62078fd1575d535bc2fa88a71c46a"
server_key_version: "encryptionkey@0"
I20260430 08:00:51.779029 19241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:51.783394 19241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:51.783936 19241 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:51.784113 19241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:51.784641 19241 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:00:51.785573 19241 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:00:51.785645 19241 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.785741 19241 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:00:51.785794 19241 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:51.794763 19241 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:37851
I20260430 08:00:51.794807 19368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:37851 every 8 connection(s)
I20260430 08:00:51.795740 19241 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:00:51.797994 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 19241
I20260430 08:00:51.798110 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:00:51.798365 13690 external_mini_cluster.cc:1468] Setting key 40b81f9eb6897090235ced0bd3802f57
I20260430 08:00:51.801605 18958 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.797717 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:60627 (local address 127.13.94.190:45303)
0430 08:00:51.797888 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.797894 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.798924 (+ 1030us) server_negotiation.cc:408] Connection header received
0430 08:00:51.799964 (+ 1040us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.799968 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.800005 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.800063 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.801225 (+ 1162us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.801228 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.801230 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.801240 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.801261 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.801268 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.801340 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.801474 (+ 134us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.801503 (+ 29us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260430 08:00:51.802296 19371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.798052 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45303 (local address 127.13.94.131:60627)
0430 08:00:51.798773 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:51.798815 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:51.799755 (+ 940us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:51.800221 (+ 466us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:51.800232 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:51.800308 (+ 76us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:51.801006 (+ 698us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:00:51.801008 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:51.801039 (+ 31us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:00:51.801088 (+ 49us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:51.801349 (+ 261us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:51.801355 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:51.801358 (+ 3us) client_negotiation.cc:770] Sending connection context
0430 08:00:51.801509 (+ 151us) client_negotiation.cc:241] Negotiation successful
0430 08:00:51.801652 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":133,"threads_started":1}
I20260430 08:00:51.803471 19369 heartbeater.cc:344] Connected to a master server at 127.13.94.190:45303
I20260430 08:00:51.803723 19369 heartbeater.cc:461] Registering TS with master...
I20260430 08:00:51.804121 19369 heartbeater.cc:507] Master 127.13.94.190:45303 requested a full tablet report, sending...
I20260430 08:00:51.805120 18919 ts_manager.cc:194] Registered new tserver with Master: 30455943b6b54ad6a00f051f9aac83fc (127.13.94.131:37851)
I20260430 08:00:51.812474 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 08:00:51.816203 18958 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.814430 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.13.94.190:45303)
0430 08:00:51.814600 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.814605 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.814740 (+ 135us) server_negotiation.cc:408] Connection header received
0430 08:00:51.814890 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.814896 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.814946 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.815033 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.815781 (+ 748us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.815785 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.815788 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.815800 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.815829 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.815837 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.815948 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.816022 (+ 74us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.816062 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260430 08:00:51.818948 18919 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:00:51.821516 18919 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:00:51.833343 19383 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.830506 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:37871 (local address 127.0.0.1:39752)
0430 08:00:51.831029 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:51.831042 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:51.831310 (+ 268us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:51.831919 (+ 609us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:51.831922 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:51.831938 (+ 16us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:51.832359 (+ 421us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:00:51.832362 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:51.832375 (+ 13us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:00:51.832399 (+ 24us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:51.833105 (+ 706us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:51.833108 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:51.833112 (+ 4us) client_negotiation.cc:770] Sending connection context
0430 08:00:51.833172 (+ 60us) client_negotiation.cc:241] Negotiation successful
0430 08:00:51.833191 (+ 19us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":71,"threads_started":1}
I20260430 08:00:51.833575 19380 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.829838 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:37851 (local address 127.0.0.1:42598)
0430 08:00:51.830240 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:51.830283 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:51.830398 (+ 115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:51.831583 (+ 1185us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:51.831589 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:51.831650 (+ 61us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:51.832218 (+ 568us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:00:51.832222 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:51.832245 (+ 23us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:00:51.832303 (+ 58us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:51.832944 (+ 641us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:51.832952 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:51.832964 (+ 12us) client_negotiation.cc:770] Sending connection context
0430 08:00:51.833108 (+ 144us) client_negotiation.cc:241] Negotiation successful
0430 08:00:51.833154 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":109,"threads_started":1}
I20260430 08:00:51.833611 19384 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.830716 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.13.94.129:33513)
0430 08:00:51.831298 (+ 582us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.831304 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.831326 (+ 22us) server_negotiation.cc:408] Connection header received
0430 08:00:51.831386 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.831390 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.831540 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.831684 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.832962 (+ 1278us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.832968 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.832975 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.832985 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.833016 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.833028 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.833139 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.833398 (+ 259us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.833457 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":140,"threads_started":1}
I20260430 08:00:51.833745 19381 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.829942 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.13.94.131:37851)
0430 08:00:51.831147 (+ 1205us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.831154 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.831179 (+ 25us) server_negotiation.cc:408] Connection header received
0430 08:00:51.831239 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.831244 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.831411 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.831569 (+ 158us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.832749 (+ 1180us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.832757 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.832768 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.832779 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.832811 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.832828 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.832987 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.833524 (+ 537us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.833589 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":110,"threads_started":1}
I20260430 08:00:51.833830 19382 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.830367 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:33513 (local address 127.0.0.1:55424)
0430 08:00:51.830678 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:51.830690 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:51.830771 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:51.832322 (+ 1551us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:51.832325 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:51.832342 (+ 17us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:51.832788 (+ 446us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:00:51.832790 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:51.832801 (+ 11us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:00:51.832824 (+ 23us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:51.833182 (+ 358us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:51.833185 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:51.833188 (+ 3us) client_negotiation.cc:770] Sending connection context
0430 08:00:51.833259 (+ 71us) client_negotiation.cc:241] Negotiation successful
0430 08:00:51.833284 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
I20260430 08:00:51.834607 19385 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.830992 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.13.94.130:37871)
0430 08:00:51.831536 (+ 544us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.831541 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.831560 (+ 19us) server_negotiation.cc:408] Connection header received
0430 08:00:51.831618 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.831625 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.831747 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.832137 (+ 390us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.832940 (+ 803us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.832945 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.832953 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.832967 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.832996 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.833007 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.833145 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.833192 (+ 47us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.833259 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":90,"threads_started":1}
I20260430 08:00:51.836548 19303 tablet_service.cc:1511] Processing CreateTablet for tablet 93e2df3affbb431e9768948869cbe7d5 (DEFAULT_TABLE table=test-table [id=0c8f0d30a676470caea0582f02995906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:51.836561 19041 tablet_service.cc:1511] Processing CreateTablet for tablet 93e2df3affbb431e9768948869cbe7d5 (DEFAULT_TABLE table=test-table [id=0c8f0d30a676470caea0582f02995906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:51.836826 19172 tablet_service.cc:1511] Processing CreateTablet for tablet 93e2df3affbb431e9768948869cbe7d5 (DEFAULT_TABLE table=test-table [id=0c8f0d30a676470caea0582f02995906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:00:51.837548 19172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e2df3affbb431e9768948869cbe7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:51.837560 19041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e2df3affbb431e9768948869cbe7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:51.837565 19303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e2df3affbb431e9768948869cbe7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:51.842942 19386 tablet_bootstrap.cc:492] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: Bootstrap starting.
I20260430 08:00:51.844291 19387 tablet_bootstrap.cc:492] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: Bootstrap starting.
I20260430 08:00:51.845008 19388 tablet_bootstrap.cc:492] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314: Bootstrap starting.
I20260430 08:00:51.845559 19386 tablet_bootstrap.cc:654] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:51.846334 19387 tablet_bootstrap.cc:654] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:51.846429 19386 log.cc:826] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:51.846920 19388 tablet_bootstrap.cc:654] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:51.847117 19387 log.cc:826] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:51.847671 19388 log.cc:826] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:51.848448 19386 tablet_bootstrap.cc:492] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: No bootstrap required, opened a new log
I20260430 08:00:51.848651 19386 ts_tablet_manager.cc:1403] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260430 08:00:51.849033 19387 tablet_bootstrap.cc:492] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: No bootstrap required, opened a new log
I20260430 08:00:51.849210 19387 ts_tablet_manager.cc:1403] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 08:00:51.849444 19388 tablet_bootstrap.cc:492] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314: No bootstrap required, opened a new log
I20260430 08:00:51.849628 19388 ts_tablet_manager.cc:1403] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 08:00:51.851907 19386 raft_consensus.cc:359] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } }
I20260430 08:00:51.852135 19386 raft_consensus.cc:385] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:51.852082 19387 raft_consensus.cc:359] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } }
I20260430 08:00:51.852211 19386 raft_consensus.cc:740] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30455943b6b54ad6a00f051f9aac83fc, State: Initialized, Role: FOLLOWER
I20260430 08:00:51.852259 19387 raft_consensus.cc:385] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:51.852308 19387 raft_consensus.cc:740] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cdebc5dedf5494d87745bf4cc206975, State: Initialized, Role: FOLLOWER
I20260430 08:00:51.852301 19388 raft_consensus.cc:359] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } }
I20260430 08:00:51.852525 19388 raft_consensus.cc:385] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:51.852615 19388 raft_consensus.cc:740] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a21e136f80b43d684283a702e4ba314, State: Initialized, Role: FOLLOWER
I20260430 08:00:51.852736 19386 consensus_queue.cc:260] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } }
I20260430 08:00:51.852731 19387 consensus_queue.cc:260] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } }
I20260430 08:00:51.853085 19388 consensus_queue.cc:260] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } }
I20260430 08:00:51.853459 19369 heartbeater.cc:499] Master 127.13.94.190:45303 was elected leader, sending a full tablet report...
I20260430 08:00:51.853606 19386 ts_tablet_manager.cc:1434] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260430 08:00:51.853753 19238 heartbeater.cc:499] Master 127.13.94.190:45303 was elected leader, sending a full tablet report...
I20260430 08:00:51.853935 19388 ts_tablet_manager.cc:1434] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260430 08:00:51.854246 19107 heartbeater.cc:499] Master 127.13.94.190:45303 was elected leader, sending a full tablet report...
I20260430 08:00:51.854691 19387 ts_tablet_manager.cc:1434] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260430 08:00:51.885783 19239 tablet.cc:2404] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:00:51.946467 19394 raft_consensus.cc:493] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:00:51.946689 19394 raft_consensus.cc:515] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } }
I20260430 08:00:51.948276 19394 leader_election.cc:290] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30455943b6b54ad6a00f051f9aac83fc (127.13.94.131:37851), 4cdebc5dedf5494d87745bf4cc206975 (127.13.94.130:37871)
I20260430 08:00:51.950471 19381 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.948731 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:60849 (local address 127.13.94.131:37851)
0430 08:00:51.948864 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.948867 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.948920 (+ 53us) server_negotiation.cc:408] Connection header received
0430 08:00:51.949001 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.949007 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.949056 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.949172 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.950096 (+ 924us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.950104 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.950106 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.950116 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.950135 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.950142 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.950237 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.950309 (+ 72us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.950342 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260430 08:00:51.950471 19109 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.948599 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:37851 (local address 127.13.94.129:60849)
0430 08:00:51.948797 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:51.948814 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:51.948905 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:51.949197 (+ 292us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:51.949200 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:51.949217 (+ 17us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:51.949940 (+ 723us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:00:51.949943 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:51.949959 (+ 16us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:00:51.949986 (+ 27us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:51.950233 (+ 247us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:51.950237 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:51.950241 (+ 4us) client_negotiation.cc:770] Sending connection context
0430 08:00:51.950315 (+ 74us) client_negotiation.cc:241] Negotiation successful
0430 08:00:51.950342 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260430 08:00:51.950963 19395 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.948990 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:37871 (local address 127.13.94.129:44243)
0430 08:00:51.949399 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:51.949432 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:51.949519 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:51.949782 (+ 263us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:51.949788 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:51.949805 (+ 17us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:00:51.950254 (+ 449us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:00:51.950257 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:00:51.950274 (+ 17us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:00:51.950297 (+ 23us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:00:51.950691 (+ 394us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:00:51.950696 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:00:51.950698 (+ 2us) client_negotiation.cc:770] Sending connection context
0430 08:00:51.950780 (+ 82us) client_negotiation.cc:241] Negotiation successful
0430 08:00:51.950808 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":142,"threads_started":1}
I20260430 08:00:51.950964 19385 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.949102 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:44243 (local address 127.13.94.130:37871)
0430 08:00:51.949236 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.949242 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.949537 (+ 295us) server_negotiation.cc:408] Connection header received
0430 08:00:51.949596 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.949599 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.949645 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.949740 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.950421 (+ 681us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.950425 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.950429 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.950444 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.950461 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.950468 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.950652 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.950805 (+ 153us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.950839 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 08:00:51.951205 19323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e2df3affbb431e9768948869cbe7d5" candidate_uuid: "3a21e136f80b43d684283a702e4ba314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30455943b6b54ad6a00f051f9aac83fc" is_pre_election: true
I20260430 08:00:51.951442 19323 raft_consensus.cc:2468] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a21e136f80b43d684283a702e4ba314 in term 0.
I20260430 08:00:51.951890 18995 leader_election.cc:304] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30455943b6b54ad6a00f051f9aac83fc, 3a21e136f80b43d684283a702e4ba314; no voters:
I20260430 08:00:51.952164 19394 raft_consensus.cc:2804] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:00:51.952237 19394 raft_consensus.cc:493] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:00:51.952270 19394 raft_consensus.cc:3060] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:51.952440 19192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e2df3affbb431e9768948869cbe7d5" candidate_uuid: "3a21e136f80b43d684283a702e4ba314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdebc5dedf5494d87745bf4cc206975" is_pre_election: true
I20260430 08:00:51.952756 19192 raft_consensus.cc:2468] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a21e136f80b43d684283a702e4ba314 in term 0.
I20260430 08:00:51.953598 19394 raft_consensus.cc:515] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } }
I20260430 08:00:51.953998 19394 leader_election.cc:290] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [CANDIDATE]: Term 1 election: Requested vote from peers 30455943b6b54ad6a00f051f9aac83fc (127.13.94.131:37851), 4cdebc5dedf5494d87745bf4cc206975 (127.13.94.130:37871)
I20260430 08:00:51.954346 19192 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e2df3affbb431e9768948869cbe7d5" candidate_uuid: "3a21e136f80b43d684283a702e4ba314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cdebc5dedf5494d87745bf4cc206975"
I20260430 08:00:51.954402 19323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93e2df3affbb431e9768948869cbe7d5" candidate_uuid: "3a21e136f80b43d684283a702e4ba314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30455943b6b54ad6a00f051f9aac83fc"
I20260430 08:00:51.954478 19192 raft_consensus.cc:3060] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:51.954541 19323 raft_consensus.cc:3060] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:51.955659 19192 raft_consensus.cc:2468] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a21e136f80b43d684283a702e4ba314 in term 1.
I20260430 08:00:51.956022 18995 leader_election.cc:304] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a21e136f80b43d684283a702e4ba314, 4cdebc5dedf5494d87745bf4cc206975; no voters:
I20260430 08:00:51.956180 19394 raft_consensus.cc:2804] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:51.956249 19323 raft_consensus.cc:2468] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a21e136f80b43d684283a702e4ba314 in term 1.
I20260430 08:00:51.956395 19394 raft_consensus.cc:697] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 1 LEADER]: Becoming Leader. State: Replica: 3a21e136f80b43d684283a702e4ba314, State: Running, Role: LEADER
I20260430 08:00:51.956771 19394 consensus_queue.cc:237] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } }
I20260430 08:00:51.959824 18919 catalog_manager.cc:5671] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a21e136f80b43d684283a702e4ba314 (127.13.94.129). New cstate: current_term: 1 leader_uuid: "3a21e136f80b43d684283a702e4ba314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a21e136f80b43d684283a702e4ba314" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 33513 } health_report { overall_health: HEALTHY } } }
W20260430 08:00:51.968861 19108 tablet.cc:2404] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:00:51.991796 19384 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:51.990354 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.13.94.129:33513)
0430 08:00:51.990497 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:51.990501 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:51.990515 (+ 14us) server_negotiation.cc:408] Connection header received
0430 08:00:51.990607 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:51.990610 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:51.990681 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:51.990751 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:51.991377 (+ 626us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:51.991381 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:51.991383 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:51.991392 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:51.991414 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:51.991421 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:51.991495 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:51.991564 (+ 69us) server_negotiation.cc:300] Negotiation successful
0430 08:00:51.991595 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 08:00:51.997365 19192 raft_consensus.cc:1275] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [term 1 FOLLOWER]: Refusing update from remote peer 3a21e136f80b43d684283a702e4ba314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:51.997398 19323 raft_consensus.cc:1275] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [term 1 FOLLOWER]: Refusing update from remote peer 3a21e136f80b43d684283a702e4ba314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:00:51.998062 19396 consensus_queue.cc:1048] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cdebc5dedf5494d87745bf4cc206975" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:51.998230 19394 consensus_queue.cc:1048] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30455943b6b54ad6a00f051f9aac83fc" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 37851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:00:52.014786 18919 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:52.015033 18919 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:00:52.017639 18919 catalog_manager.cc:5958] T 00000000000000000000000000000000 P e2a6f402da36486eaec652293468b548: Sending DeleteTablet for 3 replicas of tablet 93e2df3affbb431e9768948869cbe7d5
I20260430 08:00:52.018518 19303 tablet_service.cc:1558] Processing DeleteTablet for tablet 93e2df3affbb431e9768948869cbe7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:52 UTC) from {username='slave'} at 127.0.0.1:42598
I20260430 08:00:52.018656 19041 tablet_service.cc:1558] Processing DeleteTablet for tablet 93e2df3affbb431e9768948869cbe7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:52 UTC) from {username='slave'} at 127.0.0.1:55424
I20260430 08:00:52.019203 19408 tablet_replica.cc:333] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314: stopping tablet replica
I20260430 08:00:52.019200 19172 tablet_service.cc:1558] Processing DeleteTablet for tablet 93e2df3affbb431e9768948869cbe7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:00:52 UTC) from {username='slave'} at 127.0.0.1:39752
I20260430 08:00:52.019565 19408 raft_consensus.cc:2243] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:00:52.019747 19410 tablet_replica.cc:333] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: stopping tablet replica
I20260430 08:00:52.019982 19408 raft_consensus.cc:2272] T 93e2df3affbb431e9768948869cbe7d5 P 3a21e136f80b43d684283a702e4ba314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:52.020165 19409 tablet_replica.cc:333] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: stopping tablet replica
I20260430 08:00:52.020226 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 18979
I20260430 08:00:52.020423 19409 raft_consensus.cc:2243] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:52.020803 19409 raft_consensus.cc:2272] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:52.022493 19410 raft_consensus.cc:2243] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:00:52.022871 19410 raft_consensus.cc:2272] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:00:52.023736 19409 ts_tablet_manager.cc:1916] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:52.024085 19410 ts_tablet_manager.cc:1916] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:00:52.026091 19410 ts_tablet_manager.cc:1929] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:00:52.026203 19410 log.cc:1199] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/93e2df3affbb431e9768948869cbe7d5
I20260430 08:00:52.026499 19410 ts_tablet_manager.cc:1950] T 93e2df3affbb431e9768948869cbe7d5 P 4cdebc5dedf5494d87745bf4cc206975: Deleting consensus metadata
I20260430 08:00:52.027022 19409 ts_tablet_manager.cc:1929] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:00:52.027122 19409 log.cc:1199] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/93e2df3affbb431e9768948869cbe7d5
I20260430 08:00:52.027452 19409 ts_tablet_manager.cc:1950] T 93e2df3affbb431e9768948869cbe7d5 P 30455943b6b54ad6a00f051f9aac83fc: Deleting consensus metadata
I20260430 08:00:52.027750 18904 catalog_manager.cc:5002] TS 4cdebc5dedf5494d87745bf4cc206975 (127.13.94.130:37871): tablet 93e2df3affbb431e9768948869cbe7d5 (table test-table [id=0c8f0d30a676470caea0582f02995906]) successfully deleted
I20260430 08:00:52.028637 18904 catalog_manager.cc:5002] TS 30455943b6b54ad6a00f051f9aac83fc (127.13.94.131:37851): tablet 93e2df3affbb431e9768948869cbe7d5 (table test-table [id=0c8f0d30a676470caea0582f02995906]) successfully deleted
W20260430 08:00:52.028877 18906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.94.129:33513 (error 108)
I20260430 08:00:52.029150 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 19110
I20260430 08:00:52.030226 19382 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:52.029692 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:33513 (local address 127.0.0.1:55450)
0430 08:00:52.030050 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:52.030119 (+ 69us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.129:33513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
W20260430 08:00:52.030493 18906 catalog_manager.cc:4729] TS 3a21e136f80b43d684283a702e4ba314 (127.13.94.129:33513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93e2df3affbb431e9768948869cbe7d5: Network error: Client connection negotiation failed: client connection to 127.13.94.129:33513: connect: Connection refused (error 111)
I20260430 08:00:52.036080 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 19241
I20260430 08:00:52.042191 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 18888
2026-04-30T08:00:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (967 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260430 08:00:52.065106 13690 mini_oidc.cc:150] Starting JWKS server
I20260430 08:00:52.066519 13690 webserver.cc:492] Webserver started at https://127.0.0.1:34811/ using document root <none> and password file <none>
I20260430 08:00:52.066653 13690 mini_oidc.cc:186] Starting OIDC Discovery server
I20260430 08:00:52.066851 13690 webserver.cc:492] Webserver started at http://127.0.0.1:43341/ using document root <none> and password file <none>
2026-04-30T08:00:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:00:52Z Disabled control of system clock
I20260430 08:00:52.075397 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:41735
--webserver_interface=127.13.94.190
--webserver_port=0
--builtin_ntp_servers=127.13.94.148:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:41735
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34811/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/testchainca.cert with env {}
W20260430 08:00:52.178987 19417 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:00:52.179235 19417 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:00:52.179287 19417 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:00:52.182806 19417 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:00:52.182883 19417 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:00:52.182900 19417 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:00:52.182926 19417 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:00:52.182943 19417 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260430 08:00:52.182961 19417 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260430 08:00:52.182982 19417 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:34811/jwks.json
I20260430 08:00:52.187517 19417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:41735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:41735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34811/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:00:52.188719 19417 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:00:52.189611 19417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:00:52.195729 19423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:00:52.195717 19422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:52.195833 19417 server_base.cc:1061] running on GCE node
W20260430 08:00:52.195717 19425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:00:52.196508 19417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:00:52.197538 19417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:00:52.198714 19417 hybrid_clock.cc:648] HybridClock initialized: now 1777536052198683 us; error 46 us; skew 500 ppm
I20260430 08:00:52.200525 19417 webserver.cc:492] Webserver started at http://127.13.94.190:44791/ using document root <none> and password file <none>
I20260430 08:00:52.201174 19417 fs_manager.cc:362] Metadata directory not provided
I20260430 08:00:52.201231 19417 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:00:52.201442 19417 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:00:52.203186 19417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "26e26573b01743c9b822361d4a5a79fb"
format_stamp: "Formatted at 2026-04-30 08:00:52 on dist-test-slave-9bxp"
server_key: "4e9563434ca7a20219736cbc1f344ffe"
server_key_iv: "5d3f7c0fed20717b71a370b3a2152df4"
server_key_version: "encryptionkey@0"
I20260430 08:00:52.203693 19417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "26e26573b01743c9b822361d4a5a79fb"
format_stamp: "Formatted at 2026-04-30 08:00:52 on dist-test-slave-9bxp"
server_key: "4e9563434ca7a20219736cbc1f344ffe"
server_key_iv: "5d3f7c0fed20717b71a370b3a2152df4"
server_key_version: "encryptionkey@0"
I20260430 08:00:52.207159 19417 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:00:52.209331 19431 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:00:52.210312 19417 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:00:52.210439 19417 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "26e26573b01743c9b822361d4a5a79fb"
format_stamp: "Formatted at 2026-04-30 08:00:52 on dist-test-slave-9bxp"
server_key: "4e9563434ca7a20219736cbc1f344ffe"
server_key_iv: "5d3f7c0fed20717b71a370b3a2152df4"
server_key_version: "encryptionkey@0"
I20260430 08:00:52.210546 19417 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:00:52.228281 19417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:00:52.229031 19417 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:00:52.229235 19417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:00:52.236894 19417 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:41735
I20260430 08:00:52.236896 19483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:41735 every 8 connection(s)
I20260430 08:00:52.238113 19417 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:00:52.240968 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 19417
I20260430 08:00:52.241110 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:00:52.241391 13690 external_mini_cluster.cc:1468] Setting key 64bf4969668d882833594696351e65d4
I20260430 08:00:52.241712 19484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:00:52.248442 19484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb: Bootstrap starting.
I20260430 08:00:52.249693 19487 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:52.242728 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.13.94.190:41735)
0430 08:00:52.243361 (+ 633us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:52.243371 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:52.243405 (+ 34us) server_negotiation.cc:408] Connection header received
0430 08:00:52.244126 (+ 721us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:52.244148 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:52.244525 (+ 377us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:52.244972 (+ 447us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:00:52.245514 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:52.246295 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:52.247032 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:52.247346 (+ 314us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:52.247970 (+ 624us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:00:52.248033 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:00:52.248048 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:00:52.248453 (+ 405us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:00:52.248491 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:00:52.248505 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:00:52.248627 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:52.248901 (+ 274us) server_negotiation.cc:300] Negotiation successful
0430 08:00:52.249079 (+ 178us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":149,"threads_started":1}
I20260430 08:00:52.251016 19484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb: Neither blocks nor log segments found. Creating new log.
I20260430 08:00:52.251895 19484 log.cc:826] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb: Log is configured to *not* fsync() on all Append() calls
I20260430 08:00:52.252746 13690 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 08:00:52.254418 19484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb: No bootstrap required, opened a new log
I20260430 08:00:52.257239 19484 raft_consensus.cc:359] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e26573b01743c9b822361d4a5a79fb" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 41735 } }
I20260430 08:00:52.257459 19484 raft_consensus.cc:385] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:00:52.257539 19484 raft_consensus.cc:740] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e26573b01743c9b822361d4a5a79fb, State: Initialized, Role: FOLLOWER
I20260430 08:00:52.258013 19484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e26573b01743c9b822361d4a5a79fb" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 41735 } }
I20260430 08:00:52.258136 19484 raft_consensus.cc:399] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:00:52.258222 19484 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:00:52.258347 19484 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:00:52.259472 19484 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e26573b01743c9b822361d4a5a79fb" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 41735 } }
I20260430 08:00:52.259941 19484 leader_election.cc:304] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26e26573b01743c9b822361d4a5a79fb; no voters:
I20260430 08:00:52.260339 19484 leader_election.cc:290] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:00:52.260468 19489 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:00:52.260797 19489 raft_consensus.cc:697] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [term 1 LEADER]: Becoming Leader. State: Replica: 26e26573b01743c9b822361d4a5a79fb, State: Running, Role: LEADER
I20260430 08:00:52.261149 19489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e26573b01743c9b822361d4a5a79fb" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 41735 } }
I20260430 08:00:52.261740 19484 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:00:52.263058 19490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26e26573b01743c9b822361d4a5a79fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e26573b01743c9b822361d4a5a79fb" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 41735 } } }
I20260430 08:00:52.263244 19490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:52.263228 19491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e26573b01743c9b822361d4a5a79fb. Latest consensus state: current_term: 1 leader_uuid: "26e26573b01743c9b822361d4a5a79fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e26573b01743c9b822361d4a5a79fb" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 41735 } } }
I20260430 08:00:52.263329 19491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb [sys.catalog]: This master's current role is: LEADER
I20260430 08:00:52.263728 19498 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:00:52.266885 19498 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:00:52.272805 19498 catalog_manager.cc:1357] Generated new cluster ID: 3dd065e4a6234caaa387a87c28f29f51
I20260430 08:00:52.272989 19498 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:00:52.303680 19498 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:00:52.304667 19498 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:00:52.310743 19498 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 26e26573b01743c9b822361d4a5a79fb: Generated new TSK 0
I20260430 08:00:52.311484 19498 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 08:00:52.351132 19487 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:52.327989 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.13.94.190:41735)
0430 08:00:52.328146 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:52.328150 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:52.328284 (+ 134us) server_negotiation.cc:408] Connection header received
0430 08:00:52.328796 (+ 512us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:52.328806 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:52.328890 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:52.329019 (+ 129us) server_negotiation.cc:227] Negotiated authn=JWT
0430 08:00:52.329449 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:52.330228 (+ 779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:52.331003 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:52.331228 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:52.331319 (+ 91us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 08:00:52.350547 (+ 19228us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0430 08:00:52.350704 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
0430 08:00:52.350898 (+ 194us) server_negotiation.cc:300] Negotiation successful
0430 08:00:52.350955 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":213,"threads_started":1}
I20260430 08:00:52.362093 19487 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:00:52.358325 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.13.94.190:41735)
0430 08:00:52.358461 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0430 08:00:52.358466 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:00:52.358630 (+ 164us) server_negotiation.cc:408] Connection header received
0430 08:00:52.358836 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:00:52.358840 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:00:52.358884 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:00:52.358972 (+ 88us) server_negotiation.cc:227] Negotiated authn=JWT
0430 08:00:52.359353 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:52.359879 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:00:52.360571 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:00:52.360832 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:52.360879 (+ 47us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 08:00:52.361726 (+ 847us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0430 08:00:52.361962 (+ 236us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43022: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":46}
W20260430 08:00:52.362185 19487 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43022: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260430 08:00:52.362864 19520 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:00:52.358224 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:41735 (local address 127.0.0.1:43022)
0430 08:00:52.358543 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0430 08:00:52.358556 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:00:52.358719 (+ 163us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:00:52.358999 (+ 280us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:00:52.359002 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:00:52.359010 (+ 8us) client_negotiation.cc:190] Negotiated authn=JWT
0430 08:00:52.359238 (+ 228us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:52.359246 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:52.360031 (+ 785us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:00:52.360034 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:00:52.360453 (+ 419us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:00:52.360461 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:00:52.360589 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:00:52.360615 (+ 26us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 08:00:52.361936 (+ 1321us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0430 08:00:52.362095 (+ 159us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.94.190:41735: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":121,"threads_started":1}
I20260430 08:01:07.370769 19526 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:07.367337 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.13.94.190:41735)
0430 08:01:07.367660 (+ 323us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:07.367666 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:07.367684 (+ 18us) server_negotiation.cc:408] Connection header received
0430 08:01:07.367751 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:07.367754 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:07.367799 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:07.367900 (+ 101us) server_negotiation.cc:227] Negotiated authn=JWT
0430 08:01:07.368297 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.368860 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:07.369570 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.369762 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:07.369805 (+ 43us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 08:01:07.370422 (+ 617us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0430 08:01:07.370614 (+ 192us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56334: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
W20260430 08:01:07.370829 19525 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:07.367204 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:41735 (local address 127.0.0.1:56334)
0430 08:01:07.367508 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:07.367527 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:07.367643 (+ 116us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:07.367918 (+ 275us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:07.367921 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:07.367930 (+ 9us) client_negotiation.cc:190] Negotiated authn=JWT
0430 08:01:07.368167 (+ 237us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:07.368175 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.369008 (+ 833us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:07.369012 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:07.369440 (+ 428us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:07.369448 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.369581 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:07.369606 (+ 25us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 08:01:07.370554 (+ 948us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0430 08:01:07.370686 (+ 132us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.94.190:41735: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":119,"threads_started":1}
W20260430 08:01:07.370848 19526 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56334: JWT verification failed: token expired
W20260430 08:01:07.373447 19526 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:07.372610 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.13.94.190:41735)
0430 08:01:07.372777 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:07.372786 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:07.372886 (+ 100us) server_negotiation.cc:408] Connection header received
0430 08:01:07.372999 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:07.373003 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:07.373059 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:07.373132 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:07.373341 (+ 209us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56340: BlockingRecv error: recv got EOF from 127.0.0.1:56340 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260430 08:01:07.377353 19536 client_negotiation.cc:376] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.13.94.190:41735
W20260430 08:01:07.377980 19526 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:07.376984 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.13.94.190:41735)
0430 08:01:07.377136 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:07.377140 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:07.377311 (+ 171us) server_negotiation.cc:408] Connection header received
0430 08:01:07.377572 (+ 261us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:07.377575 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:07.377628 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:07.377697 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:07.377882 (+ 185us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56346: BlockingRecv error: recv got EOF from 127.0.0.1:56346 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260430 08:01:07.378152 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 19417
2026-04-30T08:01:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15354 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260430 08:01:07.418751 13690 mini_oidc.cc:150] Starting JWKS server
I20260430 08:01:07.419624 13690 webserver.cc:492] Webserver started at https://127.0.0.1:46401/ using document root <none> and password file <none>
I20260430 08:01:07.419706 13690 mini_oidc.cc:186] Starting OIDC Discovery server
I20260430 08:01:07.419871 13690 webserver.cc:492] Webserver started at http://127.0.0.1:46863/ using document root <none> and password file <none>
2026-04-30T08:01:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:07Z Disabled control of system clock
I20260430 08:01:07.428393 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:38179
--webserver_interface=127.13.94.190
--webserver_port=0
--builtin_ntp_servers=127.13.94.148:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:38179
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46401/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/testchainca.cert with env {}
W20260430 08:01:07.536504 19542 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:07.536829 19542 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:07.536886 19542 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:07.540267 19542 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:07.540346 19542 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:07.540369 19542 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:07.540388 19542 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:07.540406 19542 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260430 08:01:07.540424 19542 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260430 08:01:07.540439 19542 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:46401/jwks.json
I20260430 08:01:07.545323 19542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:38179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:38179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46401/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:07.546414 19542 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:07.547241 19542 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:07.553027 19550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:07.553032 19548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:07.553499 19542 server_base.cc:1061] running on GCE node
W20260430 08:01:07.553031 19547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:07.554090 19542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:07.555104 19542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:07.556320 19542 hybrid_clock.cc:648] HybridClock initialized: now 1777536067556287 us; error 49 us; skew 500 ppm
I20260430 08:01:07.558615 19542 webserver.cc:492] Webserver started at http://127.13.94.190:42959/ using document root <none> and password file <none>
I20260430 08:01:07.559257 19542 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:07.559353 19542 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:07.559594 19542 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:07.561481 19542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "283022a1c5ac499fbb0726c29b234761"
format_stamp: "Formatted at 2026-04-30 08:01:07 on dist-test-slave-9bxp"
server_key: "5aa96ac669daae9c473531faf3c99b93"
server_key_iv: "dd2b73116e20db20d3266856f3268441"
server_key_version: "encryptionkey@0"
I20260430 08:01:07.561998 19542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "283022a1c5ac499fbb0726c29b234761"
format_stamp: "Formatted at 2026-04-30 08:01:07 on dist-test-slave-9bxp"
server_key: "5aa96ac669daae9c473531faf3c99b93"
server_key_iv: "dd2b73116e20db20d3266856f3268441"
server_key_version: "encryptionkey@0"
I20260430 08:01:07.565563 19542 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260430 08:01:07.567927 19556 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:07.569077 19542 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:01:07.569217 19542 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "283022a1c5ac499fbb0726c29b234761"
format_stamp: "Formatted at 2026-04-30 08:01:07 on dist-test-slave-9bxp"
server_key: "5aa96ac669daae9c473531faf3c99b93"
server_key_iv: "dd2b73116e20db20d3266856f3268441"
server_key_version: "encryptionkey@0"
I20260430 08:01:07.569336 19542 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:07.585455 19542 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:07.586081 19542 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:07.586220 19542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:07.593096 19542 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:38179
I20260430 08:01:07.593111 19608 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:38179 every 8 connection(s)
I20260430 08:01:07.594244 19542 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:07.597332 19609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:07.603206 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761: Bootstrap starting.
I20260430 08:01:07.603987 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 19542
I20260430 08:01:07.604089 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:07.604475 13690 external_mini_cluster.cc:1468] Setting key 708340ec43f084b66d1f1bd0d9e3b1b9
I20260430 08:01:07.606139 19609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:07.606899 19609 log.cc:826] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:07.609143 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761: No bootstrap required, opened a new log
I20260430 08:01:07.612485 19609 raft_consensus.cc:359] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283022a1c5ac499fbb0726c29b234761" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38179 } }
I20260430 08:01:07.612689 19612 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:07.605735 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.13.94.190:38179)
0430 08:01:07.606412 (+ 677us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:07.606421 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:07.606454 (+ 33us) server_negotiation.cc:408] Connection header received
0430 08:01:07.607077 (+ 623us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:07.607101 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:07.607544 (+ 443us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:07.607892 (+ 348us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:07.608479 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.609364 (+ 885us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:07.610247 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.610554 (+ 307us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:07.611084 (+ 530us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:07.611105 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:07.611117 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:07.611537 (+ 420us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:07.611575 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:07.611590 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:07.611669 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:07.611868 (+ 199us) server_negotiation.cc:300] Negotiation successful
0430 08:01:07.612042 (+ 174us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":180,"threads_started":1}
I20260430 08:01:07.612748 19609 raft_consensus.cc:385] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:07.612905 19609 raft_consensus.cc:740] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283022a1c5ac499fbb0726c29b234761, State: Initialized, Role: FOLLOWER
I20260430 08:01:07.613358 19609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283022a1c5ac499fbb0726c29b234761" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38179 } }
I20260430 08:01:07.613518 19609 raft_consensus.cc:399] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:07.613626 19609 raft_consensus.cc:493] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:07.613757 19609 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:07.614892 19609 raft_consensus.cc:515] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283022a1c5ac499fbb0726c29b234761" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38179 } }
I20260430 08:01:07.615295 19609 leader_election.cc:304] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 283022a1c5ac499fbb0726c29b234761; no voters:
I20260430 08:01:07.615651 19609 leader_election.cc:290] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:07.615728 19614 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:07.615836 13690 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 08:01:07.615960 19614 raft_consensus.cc:697] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [term 1 LEADER]: Becoming Leader. State: Replica: 283022a1c5ac499fbb0726c29b234761, State: Running, Role: LEADER
I20260430 08:01:07.616235 19614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283022a1c5ac499fbb0726c29b234761" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38179 } }
I20260430 08:01:07.616891 19609 sys_catalog.cc:565] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:07.617439 19614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 283022a1c5ac499fbb0726c29b234761. Latest consensus state: current_term: 1 leader_uuid: "283022a1c5ac499fbb0726c29b234761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283022a1c5ac499fbb0726c29b234761" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38179 } } }
I20260430 08:01:07.617632 19614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:07.617844 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "283022a1c5ac499fbb0726c29b234761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283022a1c5ac499fbb0726c29b234761" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 38179 } } }
I20260430 08:01:07.617965 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:07.618398 19619 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:07.621755 19619 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:07.627409 19619 catalog_manager.cc:1357] Generated new cluster ID: a5fadaec52ae47dcbb3613528d38f529
I20260430 08:01:07.627525 19619 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:07.636457 19619 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:07.637490 19619 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:07.644413 19619 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 283022a1c5ac499fbb0726c29b234761: Generated new TSK 0
I20260430 08:01:07.645248 19619 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260430 08:01:07.672008 19639 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260430 08:01:07.672098 19639 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260430 08:01:07.672904 19612 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:07.656832 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.13.94.190:38179)
0430 08:01:07.656974 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:07.656978 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:07.657211 (+ 233us) server_negotiation.cc:408] Connection header received
0430 08:01:07.657418 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:07.657421 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:07.657475 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:07.657587 (+ 112us) server_negotiation.cc:227] Negotiated authn=JWT
0430 08:01:07.657971 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.658510 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:07.659194 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.659395 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:07.659475 (+ 80us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 08:01:07.672414 (+ 12939us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0430 08:01:07.672751 (+ 337us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260430 08:01:07.672986 19638 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:07.656736 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:38179 (local address 127.0.0.1:57348)
0430 08:01:07.657113 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:07.657125 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:07.657266 (+ 141us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:07.657591 (+ 325us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:07.657594 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:07.657603 (+ 9us) client_negotiation.cc:190] Negotiated authn=JWT
0430 08:01:07.657850 (+ 247us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:07.657858 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.658641 (+ 783us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:07.658645 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:07.659070 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:07.659078 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.659215 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:07.659240 (+ 25us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 08:01:07.672640 (+ 13400us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0430 08:01:07.672825 (+ 185us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.94.190:38179: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":105,"threads_started":1}
W20260430 08:01:07.672994 19612 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260430 08:01:07.673781 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 19542
2026-04-30T08:01:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (295 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260430 08:01:07.714250 13690 mini_oidc.cc:150] Starting JWKS server
I20260430 08:01:07.715193 13690 webserver.cc:492] Webserver started at https://127.0.0.1:34345/ using document root <none> and password file <none>
I20260430 08:01:07.715314 13690 mini_oidc.cc:186] Starting OIDC Discovery server
I20260430 08:01:07.715574 13690 webserver.cc:492] Webserver started at http://127.0.0.1:45903/ using document root <none> and password file <none>
2026-04-30T08:01:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:07Z Disabled control of system clock
I20260430 08:01:07.723716 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:37715
--webserver_interface=127.13.94.190
--webserver_port=0
--builtin_ntp_servers=127.13.94.148:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:37715
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34345/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260430 08:01:07.826694 19645 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:07.826944 19645 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:07.826993 19645 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:07.830500 19645 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:07.830577 19645 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:07.830593 19645 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:07.830612 19645 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:07.830631 19645 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260430 08:01:07.830708 19645 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260430 08:01:07.830735 19645 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:34345/jwks.json
I20260430 08:01:07.835033 19645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:37715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:37715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34345/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:07.836222 19645 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:07.837379 19645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:07.843086 19651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:07.843205 19650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:07.843205 19645 server_base.cc:1061] running on GCE node
W20260430 08:01:07.843086 19653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:07.843914 19645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:07.844938 19645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:07.846120 19645 hybrid_clock.cc:648] HybridClock initialized: now 1777536067846105 us; error 28 us; skew 500 ppm
I20260430 08:01:07.847888 19645 webserver.cc:492] Webserver started at http://127.13.94.190:37961/ using document root <none> and password file <none>
I20260430 08:01:07.848452 19645 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:07.848536 19645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:07.848814 19645 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:07.850462 19645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "b0080241a14c42499efe02b5c2de0f49"
format_stamp: "Formatted at 2026-04-30 08:01:07 on dist-test-slave-9bxp"
server_key: "73a94c7c46576fe3ce544ed7d7f10a64"
server_key_iv: "0e716a8a933b373104fed84949a456eb"
server_key_version: "encryptionkey@0"
I20260430 08:01:07.850935 19645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "b0080241a14c42499efe02b5c2de0f49"
format_stamp: "Formatted at 2026-04-30 08:01:07 on dist-test-slave-9bxp"
server_key: "73a94c7c46576fe3ce544ed7d7f10a64"
server_key_iv: "0e716a8a933b373104fed84949a456eb"
server_key_version: "encryptionkey@0"
I20260430 08:01:07.854218 19645 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:01:07.856467 19659 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:07.857450 19645 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:07.857585 19645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "b0080241a14c42499efe02b5c2de0f49"
format_stamp: "Formatted at 2026-04-30 08:01:07 on dist-test-slave-9bxp"
server_key: "73a94c7c46576fe3ce544ed7d7f10a64"
server_key_iv: "0e716a8a933b373104fed84949a456eb"
server_key_version: "encryptionkey@0"
I20260430 08:01:07.857692 19645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:07.870232 19645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:07.870916 19645 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:07.871114 19645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:07.878669 19711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:37715 every 8 connection(s)
I20260430 08:01:07.878669 19645 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:37715
I20260430 08:01:07.879843 19645 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:07.882799 19712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:07.888031 19712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49: Bootstrap starting.
I20260430 08:01:07.888926 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 19645
I20260430 08:01:07.889075 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:07.889320 13690 external_mini_cluster.cc:1468] Setting key 598366566c7d45c9e47e64fdfddb204e
I20260430 08:01:07.890724 19712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:07.891444 19712 log.cc:826] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:07.893549 19712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49: No bootstrap required, opened a new log
I20260430 08:01:07.896610 19712 raft_consensus.cc:359] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0080241a14c42499efe02b5c2de0f49" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37715 } }
I20260430 08:01:07.896903 19712 raft_consensus.cc:385] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:07.896999 19712 raft_consensus.cc:740] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0080241a14c42499efe02b5c2de0f49, State: Initialized, Role: FOLLOWER
I20260430 08:01:07.897045 19715 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:07.890396 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.13.94.190:37715)
0430 08:01:07.891030 (+ 634us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:07.891038 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:07.891069 (+ 31us) server_negotiation.cc:408] Connection header received
0430 08:01:07.891645 (+ 576us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:07.891674 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:07.892100 (+ 426us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:07.892420 (+ 320us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:07.892990 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.893801 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:07.894675 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.894877 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:07.895427 (+ 550us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:07.895448 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:07.895459 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:07.895823 (+ 364us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:07.895857 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:07.895870 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:07.895945 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:07.896137 (+ 192us) server_negotiation.cc:300] Negotiation successful
0430 08:01:07.896316 (+ 179us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":183,"threads_started":1}
I20260430 08:01:07.897538 19712 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0080241a14c42499efe02b5c2de0f49" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37715 } }
I20260430 08:01:07.897691 19712 raft_consensus.cc:399] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:07.897753 19712 raft_consensus.cc:493] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:07.897845 19712 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:07.898753 19712 raft_consensus.cc:515] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0080241a14c42499efe02b5c2de0f49" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37715 } }
I20260430 08:01:07.899129 19712 leader_election.cc:304] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0080241a14c42499efe02b5c2de0f49; no voters:
I20260430 08:01:07.899462 19712 leader_election.cc:290] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:07.899539 19717 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:07.899663 13690 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 08:01:07.899771 19717 raft_consensus.cc:697] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [term 1 LEADER]: Becoming Leader. State: Replica: b0080241a14c42499efe02b5c2de0f49, State: Running, Role: LEADER
I20260430 08:01:07.900102 19717 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0080241a14c42499efe02b5c2de0f49" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37715 } }
I20260430 08:01:07.900650 19712 sys_catalog.cc:565] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:07.901337 19718 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0080241a14c42499efe02b5c2de0f49. Latest consensus state: current_term: 1 leader_uuid: "b0080241a14c42499efe02b5c2de0f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0080241a14c42499efe02b5c2de0f49" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37715 } } }
I20260430 08:01:07.901455 19718 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:07.901955 19719 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0080241a14c42499efe02b5c2de0f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0080241a14c42499efe02b5c2de0f49" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37715 } } }
I20260430 08:01:07.902164 19719 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49 [sys.catalog]: This master's current role is: LEADER
W20260430 08:01:07.905436 19733 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 08:01:07.905501 19733 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 08:01:07.905586 19726 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:07.906347 19726 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:07.911748 19726 catalog_manager.cc:1357] Generated new cluster ID: e407e2a3f18044f5a6bc28ed431315d6
I20260430 08:01:07.911830 19726 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:07.924453 19726 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:07.925384 19726 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:07.939440 19726 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b0080241a14c42499efe02b5c2de0f49: Generated new TSK 0
I20260430 08:01:07.940218 19726 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260430 08:01:07.976445 19742 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260430 08:01:07.976534 19742 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260430 08:01:07.978132 19715 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:07.951059 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.13.94.190:37715)
0430 08:01:07.951263 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:07.951267 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:07.952911 (+ 1644us) server_negotiation.cc:408] Connection header received
0430 08:01:07.953132 (+ 221us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:07.953136 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:07.953208 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:07.953316 (+ 108us) server_negotiation.cc:227] Negotiated authn=JWT
0430 08:01:07.953873 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.954804 (+ 931us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:07.955534 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.955764 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:07.955851 (+ 87us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 08:01:07.977549 (+ 21698us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0430 08:01:07.977929 (+ 380us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260430 08:01:07.978214 19715 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260430 08:01:07.978921 19741 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:07.950947 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37715 (local address 127.0.0.1:33348)
0430 08:01:07.952810 (+ 1863us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:07.952829 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:07.952969 (+ 140us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:07.953333 (+ 364us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:07.953338 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:07.953353 (+ 15us) client_negotiation.cc:190] Negotiated authn=JWT
0430 08:01:07.953731 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:07.953742 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.954941 (+ 1199us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:07.954947 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:07.955398 (+ 451us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:07.955407 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:07.955544 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:07.955573 (+ 29us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 08:01:07.977778 (+ 22205us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0430 08:01:07.978695 (+ 917us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.94.190:37715: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34345/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":1721,"thread_start_us":119,"threads_started":1}
I20260430 08:01:07.979677 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 19645
2026-04-30T08:01:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (307 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-04-30T08:01:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:08Z Disabled control of system clock
I20260430 08:01:08.030467 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:45677
--webserver_interface=127.13.94.190
--webserver_port=0
--builtin_ntp_servers=127.13.94.148:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:45677
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/testchainca.cert with env {}
W20260430 08:01:08.134752 19746 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:08.135067 19746 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:08.135162 19746 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:08.138705 19746 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:08.138831 19746 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:08.138864 19746 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:08.138904 19746 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:08.138945 19746 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260430 08:01:08.139004 19746 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260430 08:01:08.139038 19746 flags.cc:432] Enabled experimental flag: --jwks_url=default.url
I20260430 08:01:08.143846 19746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:45677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:45677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:08.145097 19746 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:08.145999 19746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:08.151872 19752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:08.151916 19754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:08.152024 19746 server_base.cc:1061] running on GCE node
W20260430 08:01:08.152177 19751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:08.152861 19746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:08.153962 19746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:08.155161 19746 hybrid_clock.cc:648] HybridClock initialized: now 1777536068155103 us; error 85 us; skew 500 ppm
I20260430 08:01:08.157384 19746 webserver.cc:492] Webserver started at http://127.13.94.190:35497/ using document root <none> and password file <none>
I20260430 08:01:08.158015 19746 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:08.158072 19746 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:08.158288 19746 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:08.160122 19746 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "21e1b4dc843443ecaa541a32c37b1c98"
format_stamp: "Formatted at 2026-04-30 08:01:08 on dist-test-slave-9bxp"
server_key: "91992810ea70054f23f40d284f8d9498"
server_key_iv: "875d5f36cf0c068ef9c266e6d19a4c19"
server_key_version: "encryptionkey@0"
I20260430 08:01:08.160640 19746 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "21e1b4dc843443ecaa541a32c37b1c98"
format_stamp: "Formatted at 2026-04-30 08:01:08 on dist-test-slave-9bxp"
server_key: "91992810ea70054f23f40d284f8d9498"
server_key_iv: "875d5f36cf0c068ef9c266e6d19a4c19"
server_key_version: "encryptionkey@0"
I20260430 08:01:08.164311 19746 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:01:08.167238 19760 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:08.168462 19746 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260430 08:01:08.168784 19746 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "21e1b4dc843443ecaa541a32c37b1c98"
format_stamp: "Formatted at 2026-04-30 08:01:08 on dist-test-slave-9bxp"
server_key: "91992810ea70054f23f40d284f8d9498"
server_key_iv: "875d5f36cf0c068ef9c266e6d19a4c19"
server_key_version: "encryptionkey@0"
I20260430 08:01:08.168916 19746 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:08.183629 19746 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:08.184350 19746 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:08.184552 19746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:08.191592 19746 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:45677
I20260430 08:01:08.191601 19812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:45677 every 8 connection(s)
I20260430 08:01:08.192660 19746 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:08.195518 19813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:08.195950 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 19746
I20260430 08:01:08.196099 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:08.196346 13690 external_mini_cluster.cc:1468] Setting key bbb3023ac05a2f6509de270265a7beb2
I20260430 08:01:08.201280 19813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98: Bootstrap starting.
I20260430 08:01:08.203528 19813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:08.203863 19816 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:08.197607 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.13.94.190:45677)
0430 08:01:08.198188 (+ 581us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:08.198196 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:08.198228 (+ 32us) server_negotiation.cc:408] Connection header received
0430 08:01:08.198771 (+ 543us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:08.198786 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:08.199102 (+ 316us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:08.199520 (+ 418us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:08.200064 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:08.200876 (+ 812us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:08.201604 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:08.201897 (+ 293us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:08.202399 (+ 502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:08.202437 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:08.202447 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:08.202819 (+ 372us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:08.202852 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:08.202865 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:08.202975 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:08.203142 (+ 167us) server_negotiation.cc:300] Negotiation successful
0430 08:01:08.203283 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260430 08:01:08.204283 19813 log.cc:826] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:08.206308 19813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98: No bootstrap required, opened a new log
I20260430 08:01:08.206447 13690 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260430 08:01:08.209131 19813 raft_consensus.cc:359] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1b4dc843443ecaa541a32c37b1c98" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45677 } }
I20260430 08:01:08.209308 19813 raft_consensus.cc:385] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:08.209384 19813 raft_consensus.cc:740] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e1b4dc843443ecaa541a32c37b1c98, State: Initialized, Role: FOLLOWER
I20260430 08:01:08.209782 19813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1b4dc843443ecaa541a32c37b1c98" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45677 } }
I20260430 08:01:08.209921 19813 raft_consensus.cc:399] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:08.209988 19813 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:08.210086 19813 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:08.211251 19813 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1b4dc843443ecaa541a32c37b1c98" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45677 } }
I20260430 08:01:08.211750 19813 leader_election.cc:304] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21e1b4dc843443ecaa541a32c37b1c98; no voters:
I20260430 08:01:08.212117 19813 leader_election.cc:290] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:08.212275 19818 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:08.212549 19818 raft_consensus.cc:697] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [term 1 LEADER]: Becoming Leader. State: Replica: 21e1b4dc843443ecaa541a32c37b1c98, State: Running, Role: LEADER
I20260430 08:01:08.212867 19818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1b4dc843443ecaa541a32c37b1c98" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45677 } }
I20260430 08:01:08.213399 19813 sys_catalog.cc:565] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:08.214479 19819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21e1b4dc843443ecaa541a32c37b1c98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1b4dc843443ecaa541a32c37b1c98" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45677 } } }
I20260430 08:01:08.214679 19820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21e1b4dc843443ecaa541a32c37b1c98. Latest consensus state: current_term: 1 leader_uuid: "21e1b4dc843443ecaa541a32c37b1c98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e1b4dc843443ecaa541a32c37b1c98" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45677 } } }
I20260430 08:01:08.214861 19820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:08.214815 19819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:08.215746 19827 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:08.218349 19827 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:08.223927 19827 catalog_manager.cc:1357] Generated new cluster ID: 1148a48efd56448b9988e4d791e2a2ac
I20260430 08:01:08.224020 19827 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:08.247742 19827 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:08.249209 19827 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:08.259625 19827 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 21e1b4dc843443ecaa541a32c37b1c98: Generated new TSK 0
I20260430 08:01:08.260310 19827 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 08:01:08.348605 19816 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:08.273548 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.13.94.190:45677)
0430 08:01:08.273701 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:08.273706 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:08.273838 (+ 132us) server_negotiation.cc:408] Connection header received
0430 08:01:08.274002 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:08.274006 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:08.274090 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:08.274209 (+ 119us) server_negotiation.cc:227] Negotiated authn=JWT
0430 08:01:08.274582 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:08.275418 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:08.276111 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:08.276316 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:08.276414 (+ 98us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0430 08:01:08.348128 (+ 71714us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0430 08:01:08.348445 (+ 317us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39630: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260430 08:01:08.348721 19816 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39630: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260430 08:01:08.348838 19842 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:08.273429 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45677 (local address 127.0.0.1:39630)
0430 08:01:08.273733 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:08.273753 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:08.273859 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:08.274231 (+ 372us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:08.274234 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:08.274242 (+ 8us) client_negotiation.cc:190] Negotiated authn=JWT
0430 08:01:08.274465 (+ 223us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:08.274473 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:08.275554 (+ 1081us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:08.275557 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:08.275994 (+ 437us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:08.276002 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:08.276096 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:08.276119 (+ 23us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0430 08:01:08.348382 (+ 72263us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0430 08:01:08.348582 (+ 200us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.94.190:45677: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":132,"threads_started":1}
I20260430 08:01:08.349651 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 19746
2026-04-30T08:01:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (346 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (115 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:01:08 dist-test-slave-9bxp krb5kdc[19854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:01:08 dist-test-slave-9bxp krb5kdc[19854](info): set up 2 sockets
Apr 30 08:01:08 dist-test-slave-9bxp krb5kdc[19854](info): commencing operation
krb5kdc: starting...
W20260430 08:01:10.593690 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.986s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:01:10 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:01:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:10.744684 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:33795
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:33795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:10.852219 19870 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:10.852548 19870 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:10.852651 19870 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:10.856237 19870 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:10.856323 19870 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:10.856348 19870 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:10.856366 19870 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:10.856397 19870 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:10.861498 19870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:33795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:33795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:10.862807 19870 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:10.863921 19870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:10.870357 19878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:10.870335 19876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:10.870357 19875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:10.870769 19870 server_base.cc:1061] running on GCE node
I20260430 08:01:10.871361 19870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:10.872339 19870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:10.873519 19870 hybrid_clock.cc:648] HybridClock initialized: now 1777536070873494 us; error 38 us; skew 500 ppm
Apr 30 08:01:10 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536070, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:10.876386 19870 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:01:10.877607 19870 webserver.cc:492] Webserver started at http://127.13.94.190:35417/ using document root <none> and password file <none>
I20260430 08:01:10.878159 19870 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:10.878207 19870 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:10.878371 19870 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:10.880064 19870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "359aa04ff4c44e37b71710c62867a232"
format_stamp: "Formatted at 2026-04-30 08:01:10 on dist-test-slave-9bxp"
server_key: "f95c7bff64ec16bb9b023c8eedc14020"
server_key_iv: "c3af8ed7f9048daa8491e954dad141a7"
server_key_version: "encryptionkey@0"
I20260430 08:01:10.880502 19870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "359aa04ff4c44e37b71710c62867a232"
format_stamp: "Formatted at 2026-04-30 08:01:10 on dist-test-slave-9bxp"
server_key: "f95c7bff64ec16bb9b023c8eedc14020"
server_key_iv: "c3af8ed7f9048daa8491e954dad141a7"
server_key_version: "encryptionkey@0"
I20260430 08:01:10.883842 19870 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260430 08:01:10.886189 19885 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:10.887212 19870 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:01:10.887364 19870 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "359aa04ff4c44e37b71710c62867a232"
format_stamp: "Formatted at 2026-04-30 08:01:10 on dist-test-slave-9bxp"
server_key: "f95c7bff64ec16bb9b023c8eedc14020"
server_key_iv: "c3af8ed7f9048daa8491e954dad141a7"
server_key_version: "encryptionkey@0"
I20260430 08:01:10.887476 19870 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:10.900182 19870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:10.906997 19870 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:10.907271 19870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:10.916185 19937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:33795 every 8 connection(s)
I20260430 08:01:10.916182 19870 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:33795
I20260430 08:01:10.917418 19870 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:10.920323 19938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:10.920809 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 19870
I20260430 08:01:10.920939 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:10.921247 13690 external_mini_cluster.cc:1468] Setting key d37651d54ec63c91b12816a4c7eb6a0a
I20260430 08:01:10.926681 19938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232: Bootstrap starting.
Apr 30 08:01:10 dist-test-slave-9bxp krb5kdc[19854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:10.929199 19938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:10.930034 19938 log.cc:826] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:10.932561 19938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232: No bootstrap required, opened a new log
I20260430 08:01:10.934903 19941 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:10.922636 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.13.94.190:33795)
0430 08:01:10.923123 (+ 487us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:10.923133 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:10.923168 (+ 35us) server_negotiation.cc:408] Connection header received
0430 08:01:10.923906 (+ 738us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:10.923925 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:10.924237 (+ 312us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:10.924594 (+ 357us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:10.925524 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:10.926278 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:10.927024 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:10.927324 (+ 300us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:10.929657 (+ 2333us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:10.929675 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:10.929687 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:10.929717 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:10.932220 (+ 2503us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:10.932670 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:10.932676 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:10.932682 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:10.932794 (+ 112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:10.933057 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:10.933060 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:10.933062 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:10.933463 (+ 401us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:10.933645 (+ 182us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:10.933901 (+ 256us) server_negotiation.cc:300] Negotiation successful
0430 08:01:10.934156 (+ 255us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":156,"threads_started":1}
I20260430 08:01:10.935930 19938 raft_consensus.cc:359] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359aa04ff4c44e37b71710c62867a232" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33795 } }
I20260430 08:01:10.936156 19938 raft_consensus.cc:385] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:10.936241 19938 raft_consensus.cc:740] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359aa04ff4c44e37b71710c62867a232, State: Initialized, Role: FOLLOWER
I20260430 08:01:10.936641 19938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359aa04ff4c44e37b71710c62867a232" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33795 } }
I20260430 08:01:10.936836 19938 raft_consensus.cc:399] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:10.937079 19938 raft_consensus.cc:493] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:10.937268 19938 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:10.938290 19938 raft_consensus.cc:515] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359aa04ff4c44e37b71710c62867a232" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33795 } }
I20260430 08:01:10.938598 19938 leader_election.cc:304] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 359aa04ff4c44e37b71710c62867a232; no voters:
I20260430 08:01:10.938932 19938 leader_election.cc:290] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:10.939157 19943 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:10.939438 19943 raft_consensus.cc:697] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [term 1 LEADER]: Becoming Leader. State: Replica: 359aa04ff4c44e37b71710c62867a232, State: Running, Role: LEADER
I20260430 08:01:10.939795 19943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359aa04ff4c44e37b71710c62867a232" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33795 } }
I20260430 08:01:10.940156 19938 sys_catalog.cc:565] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:10.941532 19945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359aa04ff4c44e37b71710c62867a232. Latest consensus state: current_term: 1 leader_uuid: "359aa04ff4c44e37b71710c62867a232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359aa04ff4c44e37b71710c62867a232" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33795 } } }
I20260430 08:01:10.941677 19945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:10.941532 19944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "359aa04ff4c44e37b71710c62867a232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359aa04ff4c44e37b71710c62867a232" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 33795 } } }
I20260430 08:01:10.942003 19944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:10.942194 19952 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:10.945843 19952 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:10.951541 19952 catalog_manager.cc:1357] Generated new cluster ID: 5bf17715d48d4a60a06209ca4951d17a
I20260430 08:01:10.951647 19952 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:10.992499 19952 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:10.993808 19952 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:11.006716 19952 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 359aa04ff4c44e37b71710c62867a232: Generated new TSK 0
I20260430 08:01:11.007490 19952 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:11.068750 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:33795
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:11.181213 19966 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:11.181452 19966 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:11.181511 19966 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:11.185102 19966 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:11.185177 19966 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:11.185299 19966 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:01:11.189903 19966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:33795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:11.191109 19966 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:11.192132 19966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:11.198839 19971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:11.198849 19974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:11.198933 19966 server_base.cc:1061] running on GCE node
W20260430 08:01:11.198836 19972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:11.199631 19966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:11.200213 19966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:11.201414 19966 hybrid_clock.cc:648] HybridClock initialized: now 1777536071201384 us; error 44 us; skew 500 ppm
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.204330 19966 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:01:11.205444 19966 webserver.cc:492] Webserver started at http://127.13.94.129:41543/ using document root <none> and password file <none>
I20260430 08:01:11.206015 19966 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:11.206065 19966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:11.206275 19966 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:11.207978 19966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "31851ece85854aa38b0943c64031a5b3"
format_stamp: "Formatted at 2026-04-30 08:01:11 on dist-test-slave-9bxp"
server_key: "472bf2c0646f8e3c92edfa1c34be94de"
server_key_iv: "5d86f1af339c0f01e2eaf7f1ebb64d8d"
server_key_version: "encryptionkey@0"
I20260430 08:01:11.208464 19966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "31851ece85854aa38b0943c64031a5b3"
format_stamp: "Formatted at 2026-04-30 08:01:11 on dist-test-slave-9bxp"
server_key: "472bf2c0646f8e3c92edfa1c34be94de"
server_key_iv: "5d86f1af339c0f01e2eaf7f1ebb64d8d"
server_key_version: "encryptionkey@0"
I20260430 08:01:11.211846 19966 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260430 08:01:11.214241 19981 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:11.215394 19966 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:01:11.215531 19966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "31851ece85854aa38b0943c64031a5b3"
format_stamp: "Formatted at 2026-04-30 08:01:11 on dist-test-slave-9bxp"
server_key: "472bf2c0646f8e3c92edfa1c34be94de"
server_key_iv: "5d86f1af339c0f01e2eaf7f1ebb64d8d"
server_key_version: "encryptionkey@0"
I20260430 08:01:11.215641 19966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:11.229362 19966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:11.232304 19966 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:11.232510 19966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:11.233373 19966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:11.234302 19966 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:11.234377 19966 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:11.234450 19966 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:11.234493 19966 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:11.244132 19966 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:35527
I20260430 08:01:11.244161 20094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:35527 every 8 connection(s)
I20260430 08:01:11.245270 19966 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:11.255393 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 19966
I20260430 08:01:11.255504 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:01:11.255812 13690 external_mini_cluster.cc:1468] Setting key 6d01d8ea4e45a416b8c7d0361e94bef4
I20260430 08:01:11.259371 19941 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:11.247401 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:38657 (local address 127.13.94.190:33795)
0430 08:01:11.247562 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:11.247566 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:11.248399 (+ 833us) server_negotiation.cc:408] Connection header received
0430 08:01:11.249420 (+ 1021us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:11.249424 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:11.249489 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:11.249600 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:11.251327 (+ 1727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.251849 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:11.252568 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.252775 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:11.255753 (+ 2978us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:11.255778 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:11.255781 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:11.255810 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:11.257402 (+ 1592us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.257918 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.257922 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.257924 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.257977 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.258292 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.258296 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.258298 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.258453 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:11.258560 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:11.259062 (+ 502us) server_negotiation.cc:300] Negotiation successful
0430 08:01:11.259199 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260430 08:01:11.260025 20097 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:11.247697 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:33795 (local address 127.13.94.129:38657)
0430 08:01:11.248233 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:11.248272 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:11.249179 (+ 907us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:11.249782 (+ 603us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:11.249791 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:11.250252 (+ 461us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:11.251134 (+ 882us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:11.251149 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.252012 (+ 863us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:11.252018 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:11.252438 (+ 420us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:11.252445 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.252719 (+ 274us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:11.253551 (+ 832us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:11.253572 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:11.255531 (+ 1959us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:11.257534 (+ 2003us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:11.257541 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:11.257556 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:11.257811 (+ 255us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:11.258086 (+ 275us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:11.258089 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:11.258090 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:11.258192 (+ 102us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:11.258561 (+ 369us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:11.258568 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:11.258808 (+ 240us) client_negotiation.cc:770] Sending connection context
0430 08:01:11.259025 (+ 217us) client_negotiation.cc:241] Negotiation successful
0430 08:01:11.259274 (+ 249us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":119,"threads_started":1}
I20260430 08:01:11.261313 20095 heartbeater.cc:344] Connected to a master server at 127.13.94.190:33795
I20260430 08:01:11.261579 20095 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:11.262126 20095 heartbeater.cc:507] Master 127.13.94.190:33795 requested a full tablet report, sending...
I20260430 08:01:11.263803 19902 ts_manager.cc:194] Registered new tserver with Master: 31851ece85854aa38b0943c64031a5b3 (127.13.94.129:35527)
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
I20260430 08:01:11.265136 19902 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:38657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:11.309494 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:33795
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:11.413781 20102 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:11.414003 20102 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:11.414108 20102 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:11.417522 20102 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:11.417589 20102 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:11.417738 20102 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:01:11.422101 20102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:33795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:11.423225 20102 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:11.424096 20102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:11.430974 20107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:11.430961 20110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:11.430964 20108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:11.431103 20102 server_base.cc:1061] running on GCE node
I20260430 08:01:11.431778 20102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:11.432359 20102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:11.433538 20102 hybrid_clock.cc:648] HybridClock initialized: now 1777536071433525 us; error 25 us; skew 500 ppm
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.436272 20102 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:01:11.437481 20102 webserver.cc:492] Webserver started at http://127.13.94.130:45369/ using document root <none> and password file <none>
I20260430 08:01:11.438071 20102 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:11.438148 20102 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:11.438357 20102 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:11.440101 20102 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "ff7fa232949a4881ae881705f373e19e"
format_stamp: "Formatted at 2026-04-30 08:01:11 on dist-test-slave-9bxp"
server_key: "a2ab95772d998e41c23c9be17ff1f255"
server_key_iv: "b6d2cd5aab2b02397831b2c02b83317e"
server_key_version: "encryptionkey@0"
I20260430 08:01:11.440606 20102 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "ff7fa232949a4881ae881705f373e19e"
format_stamp: "Formatted at 2026-04-30 08:01:11 on dist-test-slave-9bxp"
server_key: "a2ab95772d998e41c23c9be17ff1f255"
server_key_iv: "b6d2cd5aab2b02397831b2c02b83317e"
server_key_version: "encryptionkey@0"
I20260430 08:01:11.444336 20102 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:01:11.446645 20117 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:11.447645 20102 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:11.447774 20102 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "ff7fa232949a4881ae881705f373e19e"
format_stamp: "Formatted at 2026-04-30 08:01:11 on dist-test-slave-9bxp"
server_key: "a2ab95772d998e41c23c9be17ff1f255"
server_key_iv: "b6d2cd5aab2b02397831b2c02b83317e"
server_key_version: "encryptionkey@0"
I20260430 08:01:11.447880 20102 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:11.473301 20102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:11.476310 20102 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:11.476518 20102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:11.477165 20102 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:11.478037 20102 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:11.478118 20102 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:11.478189 20102 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:11.478243 20102 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:11.487675 20102 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:38763
I20260430 08:01:11.487738 20230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:38763 every 8 connection(s)
I20260430 08:01:11.488656 20102 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:01:11.495831 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 20102
I20260430 08:01:11.495980 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:01:11.496268 13690 external_mini_cluster.cc:1468] Setting key 8881bf5d07b3a46be816b1cb55dbd87f
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:11.502352 19941 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:11.490563 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:56203 (local address 127.13.94.190:33795)
0430 08:01:11.490713 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:11.490717 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:11.491418 (+ 701us) server_negotiation.cc:408] Connection header received
0430 08:01:11.492312 (+ 894us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:11.492316 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:11.492368 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:11.492444 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:11.494314 (+ 1870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.494817 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:11.495560 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.495787 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:11.498184 (+ 2397us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:11.498205 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:11.498211 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:11.498246 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:11.499990 (+ 1744us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.500690 (+ 700us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.500693 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.500745 (+ 52us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.500808 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.501188 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.501191 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.501192 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.501347 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:11.501428 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:11.502110 (+ 682us) server_negotiation.cc:300] Negotiation successful
0430 08:01:11.502228 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 08:01:11.503342 20233 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:11.490830 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:33795 (local address 127.13.94.130:56203)
0430 08:01:11.491252 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:11.491285 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:11.492114 (+ 829us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:11.492639 (+ 525us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:11.492648 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:11.493169 (+ 521us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:11.494123 (+ 954us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:11.494137 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.494952 (+ 815us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:11.494955 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:11.495381 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:11.495388 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.495557 (+ 169us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:11.496244 (+ 687us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:11.496262 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:11.498022 (+ 1760us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:11.500167 (+ 2145us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:11.500177 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:11.500191 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:11.500559 (+ 368us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:11.500907 (+ 348us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:11.500913 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:11.500918 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:11.501084 (+ 166us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:11.501462 (+ 378us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:11.501471 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:11.501786 (+ 315us) client_negotiation.cc:770] Sending connection context
0430 08:01:11.502079 (+ 293us) client_negotiation.cc:241] Negotiation successful
0430 08:01:11.502351 (+ 272us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":100,"threads_started":1}
I20260430 08:01:11.504542 20231 heartbeater.cc:344] Connected to a master server at 127.13.94.190:33795
I20260430 08:01:11.504854 20231 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:11.505419 20231 heartbeater.cc:507] Master 127.13.94.190:33795 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
I20260430 08:01:11.506664 19902 ts_manager.cc:194] Registered new tserver with Master: ff7fa232949a4881ae881705f373e19e (127.13.94.130:38763)
I20260430 08:01:11.507293 19902 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:56203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:11.551411 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:33795
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:11.657423 20238 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:11.657647 20238 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:11.657753 20238 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:11.661546 20238 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:11.661623 20238 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:11.661741 20238 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:01:11.667346 20238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:33795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:11.668517 20238 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:11.669481 20238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:11.675928 20246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:11.675923 20243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:11.676201 20238 server_base.cc:1061] running on GCE node
W20260430 08:01:11.675923 20244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:11.676748 20238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:11.677290 20238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:11.678463 20238 hybrid_clock.cc:648] HybridClock initialized: now 1777536071678438 us; error 37 us; skew 500 ppm
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.681242 20238 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:01:11.682323 20238 webserver.cc:492] Webserver started at http://127.13.94.131:37013/ using document root <none> and password file <none>
I20260430 08:01:11.682885 20238 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:11.682934 20238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:11.683147 20238 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:11.684895 20238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "f1258d45f47c425ba8739caff7766f34"
format_stamp: "Formatted at 2026-04-30 08:01:11 on dist-test-slave-9bxp"
server_key: "795f7ae089e77dbd03ae35a5bb5000aa"
server_key_iv: "43a9e4ee50480f31b1a05b74637c34dd"
server_key_version: "encryptionkey@0"
I20260430 08:01:11.685416 20238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "f1258d45f47c425ba8739caff7766f34"
format_stamp: "Formatted at 2026-04-30 08:01:11 on dist-test-slave-9bxp"
server_key: "795f7ae089e77dbd03ae35a5bb5000aa"
server_key_iv: "43a9e4ee50480f31b1a05b74637c34dd"
server_key_version: "encryptionkey@0"
I20260430 08:01:11.688663 20238 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:01:11.690919 20253 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:11.692103 20238 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:11.692221 20238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "f1258d45f47c425ba8739caff7766f34"
format_stamp: "Formatted at 2026-04-30 08:01:11 on dist-test-slave-9bxp"
server_key: "795f7ae089e77dbd03ae35a5bb5000aa"
server_key_iv: "43a9e4ee50480f31b1a05b74637c34dd"
server_key_version: "encryptionkey@0"
I20260430 08:01:11.692322 20238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:11.701861 20238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:11.705017 20238 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:11.705231 20238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:11.705817 20238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:11.706774 20238 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:11.706822 20238 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:11.706893 20238 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:11.706928 20238 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:11.716223 20238 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:36763
I20260430 08:01:11.716244 20366 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:36763 every 8 connection(s)
I20260430 08:01:11.717211 20238 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:01:11.717739 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 20238
I20260430 08:01:11.717854 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:01:11.718112 13690 external_mini_cluster.cc:1468] Setting key 537550caa3cd579729841f8f917a2a80
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:11.729489 19941 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:11.719172 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:42229 (local address 127.13.94.190:33795)
0430 08:01:11.719406 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:11.719411 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:11.720007 (+ 596us) server_negotiation.cc:408] Connection header received
0430 08:01:11.720981 (+ 974us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:11.720985 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:11.721052 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:11.721118 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:11.722642 (+ 1524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.723233 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:11.723868 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.724012 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:11.726186 (+ 2174us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:11.726213 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:11.726216 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:11.726242 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:11.727624 (+ 1382us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.728119 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.728123 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.728124 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.728168 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.728465 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.728469 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.728471 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.728613 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:11.728746 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:11.729208 (+ 462us) server_negotiation.cc:300] Negotiation successful
0430 08:01:11.729324 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260430 08:01:11.730114 20369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:11.719422 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:33795 (local address 127.13.94.131:42229)
0430 08:01:11.719888 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:11.719928 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:11.720800 (+ 872us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:11.721249 (+ 449us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:11.721257 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:11.721611 (+ 354us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:11.722474 (+ 863us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:11.722488 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.723359 (+ 871us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:11.723362 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:11.723749 (+ 387us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:11.723755 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.723906 (+ 151us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:11.724447 (+ 541us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:11.724464 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:11.726029 (+ 1565us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:11.727755 (+ 1726us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:11.727761 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:11.727773 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:11.728001 (+ 228us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:11.728268 (+ 267us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:11.728275 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:11.728277 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:11.728370 (+ 93us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:11.728743 (+ 373us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:11.728748 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:11.728962 (+ 214us) client_negotiation.cc:770] Sending connection context
0430 08:01:11.729162 (+ 200us) client_negotiation.cc:241] Negotiation successful
0430 08:01:11.729398 (+ 236us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":144,"threads_started":1}
I20260430 08:01:11.731477 20367 heartbeater.cc:344] Connected to a master server at 127.13.94.190:33795
I20260430 08:01:11.731789 20367 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:11.732447 20367 heartbeater.cc:507] Master 127.13.94.190:33795 requested a full tablet report, sending...
I20260430 08:01:11.733525 19902 ts_manager.cc:194] Registered new tserver with Master: f1258d45f47c425ba8739caff7766f34 (127.13.94.131:36763)
I20260430 08:01:11.734036 19902 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:42229
I20260430 08:01:11.744318 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/test-admin.keytab.
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.761718 13690 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260430 08:01:11.764624 20374 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:11.763168 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.13.94.129:35527)
0430 08:01:11.763444 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:11.763450 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:11.763493 (+ 43us) server_negotiation.cc:408] Connection header received
0430 08:01:11.763546 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:11.763550 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:11.763783 (+ 233us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:11.763936 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:11.764437 (+ 501us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54790: BlockingRecv error: recv got EOF from 127.0.0.1:54790 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
W20260430 08:01:11.765162 13690 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5cc)
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.766151 13690 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.767076 13690 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:01:11.776124 20374 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:11.767966 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.13.94.129:35527)
0430 08:01:11.768100 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:11.768107 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:11.768171 (+ 64us) server_negotiation.cc:408] Connection header received
0430 08:01:11.768322 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:11.768325 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:11.768378 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:11.768467 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:11.769214 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.769912 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:11.770592 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.770856 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:11.772749 (+ 1893us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:11.772779 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:11.772791 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:11.772815 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:11.774435 (+ 1620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.774896 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.774902 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.774903 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.774950 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.775202 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.775205 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.775207 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.775392 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:11.775622 (+ 230us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:11.775819 (+ 197us) server_negotiation.cc:300] Negotiation successful
0430 08:01:11.775966 (+ 147us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260430 08:01:11.779922 20374 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:11.779010 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.13.94.129:35527)
0430 08:01:11.779175 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:11.779178 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:11.779191 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:11.779291 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:11.779293 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:11.779332 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:11.779397 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:11.779807 (+ 410us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54800: BlockingRecv error: recv got EOF from 127.0.0.1:54800 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260430 08:01:11.780009 13690 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5cc' of type FILE
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.781137 13690 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.782115 13690 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:01:11.789974 20374 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:11.782927 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.13.94.129:35527)
0430 08:01:11.783044 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:11.783047 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:11.783067 (+ 20us) server_negotiation.cc:408] Connection header received
0430 08:01:11.783166 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:11.783168 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:11.783203 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:11.783266 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:11.783848 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.784323 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:11.784984 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.785193 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:11.787080 (+ 1887us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:11.787122 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:11.787124 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:11.787149 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:11.788557 (+ 1408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.788956 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.788959 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.788960 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.789003 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.789271 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.789273 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.789275 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.789392 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:11.789485 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:11.789751 (+ 266us) server_negotiation.cc:300] Negotiation successful
0430 08:01:11.789851 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260430 08:01:11.793071 20374 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:11.792005 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.13.94.129:35527)
0430 08:01:11.792146 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:11.792150 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:11.792272 (+ 122us) server_negotiation.cc:408] Connection header received
0430 08:01:11.792417 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:11.792421 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:11.792464 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:11.792551 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:11.792973 (+ 422us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54812: BlockingRecv error: recv got EOF from 127.0.0.1:54812 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260430 08:01:11.793290 13690 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777536011314967-13690-0/krb5kdc/krb5cc' of type FILE
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.794313 13690 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:11.795198 13690 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[19854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.129@KRBTEST.COM
I20260430 08:01:11.803275 20374 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:11.796044 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.13.94.129:35527)
0430 08:01:11.796203 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:11.796207 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:11.796278 (+ 71us) server_negotiation.cc:408] Connection header received
0430 08:01:11.796378 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:11.796380 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:11.796416 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:11.796480 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:11.797169 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.797627 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:11.798506 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:11.798722 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:11.800651 (+ 1929us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:11.800666 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:11.800669 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:11.800691 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:11.801972 (+ 1281us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.802369 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.802372 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.802373 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.802414 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:11.802634 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:11.802637 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:11.802638 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:11.802751 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:11.802849 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:11.803059 (+ 210us) server_negotiation.cc:300] Negotiation successful
0430 08:01:11.803152 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260430 08:01:11.804368 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 19966
I20260430 08:01:11.810742 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 20102
I20260430 08:01:11.816344 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 20238
I20260430 08:01:11.821672 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 19870
2026-04-30T08:01:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3236 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[20387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[20387](info): set up 2 sockets
Apr 30 08:01:11 dist-test-slave-9bxp krb5kdc[20387](info): commencing operation
krb5kdc: starting...
W20260430 08:01:13.830044 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.986s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:01:13 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:01:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:13.980616 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:42167
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:42167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:14.084028 20403 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:14.084285 20403 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:14.084342 20403 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:14.087693 20403 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:14.087774 20403 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:14.087797 20403 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:14.087816 20403 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:14.087834 20403 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:14.092413 20403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:42167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:42167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:14.093564 20403 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:14.094419 20403 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:14.100327 20409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:14.100353 20411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:14.100327 20408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:14.100767 20403 server_base.cc:1061] running on GCE node
I20260430 08:01:14.101346 20403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:14.102303 20403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:14.103497 20403 hybrid_clock.cc:648] HybridClock initialized: now 1777536074103478 us; error 39 us; skew 500 ppm
Apr 30 08:01:14 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536074, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:14.106348 20403 init.cc:377] Logged in from keytab as oryx/127.13.94.190@KRBTEST.COM (short username oryx)
I20260430 08:01:14.107468 20403 webserver.cc:492] Webserver started at http://127.13.94.190:46603/ using document root <none> and password file <none>
I20260430 08:01:14.108068 20403 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:14.108146 20403 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:14.108354 20403 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:14.110118 20403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "15f647c7facb44c89330f7899a1f3c99"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "7108c6fdd710d193929271229860b1db"
server_key_iv: "ff0afd7e906a9023e4f97556aef8853e"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.110610 20403 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "15f647c7facb44c89330f7899a1f3c99"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "7108c6fdd710d193929271229860b1db"
server_key_iv: "ff0afd7e906a9023e4f97556aef8853e"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.114043 20403 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260430 08:01:14.116310 20418 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.117364 20403 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:01:14.117518 20403 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "15f647c7facb44c89330f7899a1f3c99"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "7108c6fdd710d193929271229860b1db"
server_key_iv: "ff0afd7e906a9023e4f97556aef8853e"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.117631 20403 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:14.133687 20403 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:14.137029 20403 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:14.137239 20403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:14.145009 20470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:42167 every 8 connection(s)
I20260430 08:01:14.145018 20403 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:42167
I20260430 08:01:14.146147 20403 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:14.146359 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 20403
I20260430 08:01:14.146502 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:14.146800 13690 external_mini_cluster.cc:1468] Setting key 5b22ecd7fd3afbb9b8b85b08b24a9bf1
I20260430 08:01:14.149636 20471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:14.155308 20471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: Bootstrap starting.
Apr 30 08:01:14 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:14.157527 20471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:14.158268 20471 log.cc:826] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:14.160025 20471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: No bootstrap required, opened a new log
I20260430 08:01:14.160674 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:14.148155 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.13.94.190:42167)
0430 08:01:14.148539 (+ 384us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:14.148550 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:14.148588 (+ 38us) server_negotiation.cc:408] Connection header received
0430 08:01:14.149420 (+ 832us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:14.149445 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:14.149800 (+ 355us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:14.150146 (+ 346us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:14.150966 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.152676 (+ 1710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:14.153414 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.153686 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:14.156007 (+ 2321us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:14.156064 (+ 57us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:14.156076 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:14.156105 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:14.157996 (+ 1891us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:14.158497 (+ 501us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:14.158509 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:14.158513 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:14.158595 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:14.158903 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:14.158909 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:14.158911 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:14.159224 (+ 313us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:14.159416 (+ 192us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:14.159838 (+ 422us) server_negotiation.cc:300] Negotiation successful
0430 08:01:14.160066 (+ 228us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":100,"threads_started":1}
I20260430 08:01:14.162930 20471 raft_consensus.cc:359] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } }
I20260430 08:01:14.163102 20471 raft_consensus.cc:385] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:14.163143 20471 raft_consensus.cc:740] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15f647c7facb44c89330f7899a1f3c99, State: Initialized, Role: FOLLOWER
I20260430 08:01:14.163549 20471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } }
I20260430 08:01:14.163682 20471 raft_consensus.cc:399] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:14.163741 20471 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:14.163851 20471 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:14.165282 20471 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } }
I20260430 08:01:14.165652 20471 leader_election.cc:304] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15f647c7facb44c89330f7899a1f3c99; no voters:
I20260430 08:01:14.166013 20471 leader_election.cc:290] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:14.166128 20476 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:14.166386 20476 raft_consensus.cc:697] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 1 LEADER]: Becoming Leader. State: Replica: 15f647c7facb44c89330f7899a1f3c99, State: Running, Role: LEADER
I20260430 08:01:14.166741 20476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } }
I20260430 08:01:14.167150 20471 sys_catalog.cc:565] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:14.168390 20478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15f647c7facb44c89330f7899a1f3c99. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } }
I20260430 08:01:14.168533 20478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:14.168380 20477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } }
I20260430 08:01:14.168717 20477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
W20260430 08:01:14.172101 20491 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 08:01:14.172197 20491 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 08:01:14.172286 20485 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:14.173105 20485 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:14.178730 20485 catalog_manager.cc:1357] Generated new cluster ID: 85a9016ecce6489d9d2fa6d395880450
I20260430 08:01:14.178817 20485 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:14.189970 20485 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:14.191152 20485 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:14.206192 20485 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: Generated new TSK 0
I20260430 08:01:14.206892 20485 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:14.264442 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:42167
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:14.373050 20499 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:14.373288 20499 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:14.373345 20499 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:14.376590 20499 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:14.376654 20499 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:14.376753 20499 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:01:14.381095 20499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:42167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:14.382102 20499 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:14.382933 20499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:14.389259 20505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:14.389256 20504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:14.389268 20507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:14.389555 20499 server_base.cc:1061] running on GCE node
I20260430 08:01:14.390069 20499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:14.390594 20499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:14.391769 20499 hybrid_clock.cc:648] HybridClock initialized: now 1777536074391753 us; error 34 us; skew 500 ppm
Apr 30 08:01:14 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536074, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:14.394358 20499 init.cc:377] Logged in from keytab as oryx/127.13.94.129@KRBTEST.COM (short username oryx)
I20260430 08:01:14.395387 20499 webserver.cc:492] Webserver started at http://127.13.94.129:40865/ using document root <none> and password file <none>
I20260430 08:01:14.395936 20499 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:14.396009 20499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:14.396219 20499 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:14.398053 20499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "fb3b88c84b80445f96a8debdb48b5fdb"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "5254d35c2688935e11458787ccd7773a"
server_key_iv: "99220912bad0f1b99e4aa4f5306961f7"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.398532 20499 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "fb3b88c84b80445f96a8debdb48b5fdb"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "5254d35c2688935e11458787ccd7773a"
server_key_iv: "99220912bad0f1b99e4aa4f5306961f7"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.401783 20499 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260430 08:01:14.403951 20514 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.404979 20499 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:01:14.405103 20499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "fb3b88c84b80445f96a8debdb48b5fdb"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "5254d35c2688935e11458787ccd7773a"
server_key_iv: "99220912bad0f1b99e4aa4f5306961f7"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.405207 20499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:14.422842 20499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:14.425567 20499 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:14.425776 20499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:14.426353 20499 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:14.427248 20499 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:14.427317 20499 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.427387 20499 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:14.427438 20499 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.437332 20499 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:36283
I20260430 08:01:14.437368 20627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:36283 every 8 connection(s)
I20260430 08:01:14.438309 20499 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:01:14.440806 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 20499
I20260430 08:01:14.440943 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:01:14.441237 13690 external_mini_cluster.cc:1468] Setting key 787ef9760ca2b9743b6fadade6fd5d10
Apr 30 08:01:14 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536074, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.94.130@KRBTEST.COM; defaulting to no policy
I20260430 08:01:14.451597 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:14.440135 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:51437 (local address 127.13.94.190:42167)
0430 08:01:14.440295 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:14.440300 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:14.441088 (+ 788us) server_negotiation.cc:408] Connection header received
0430 08:01:14.442033 (+ 945us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:14.442037 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:14.442097 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:14.442203 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:14.443350 (+ 1147us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.443868 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:14.444530 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.444751 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:14.447943 (+ 3192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:14.447968 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:14.447970 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:14.447998 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:14.449556 (+ 1558us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:14.450123 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:14.450126 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:14.450127 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:14.450171 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:14.450472 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:14.450476 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:14.450478 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:14.450640 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:14.450751 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:14.451276 (+ 525us) server_negotiation.cc:300] Negotiation successful
0430 08:01:14.451405 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 08:01:14.452298 20630 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:14.440463 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.129:51437)
0430 08:01:14.440934 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:14.440975 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:14.441834 (+ 859us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:14.442373 (+ 539us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:14.442381 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:14.442754 (+ 373us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:14.443206 (+ 452us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:14.443216 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.444010 (+ 794us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:14.444013 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:14.444397 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:14.444403 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.444666 (+ 263us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:14.445845 (+ 1179us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:14.445863 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:14.447766 (+ 1903us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:14.449685 (+ 1919us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:14.449691 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:14.449704 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:14.449955 (+ 251us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:14.450276 (+ 321us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:14.450282 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:14.450288 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:14.450376 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:14.450752 (+ 376us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:14.450759 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:14.450992 (+ 233us) client_negotiation.cc:770] Sending connection context
0430 08:01:14.451215 (+ 223us) client_negotiation.cc:241] Negotiation successful
0430 08:01:14.451441 (+ 226us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.130@KRBTEST.COM" created.
I20260430 08:01:14.453691 20628 heartbeater.cc:344] Connected to a master server at 127.13.94.190:42167
I20260430 08:01:14.453975 20628 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:14.454500 20628 heartbeater.cc:507] Master 127.13.94.190:42167 requested a full tablet report, sending...
I20260430 08:01:14.456306 20435 ts_manager.cc:194] Registered new tserver with Master: fb3b88c84b80445f96a8debdb48b5fdb (127.13.94.129:36283)
I20260430 08:01:14.458084 20435 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:51437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:14.496512 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:42167
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:14.600138 20635 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:14.600374 20635 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:14.600472 20635 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:14.603924 20635 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:14.603991 20635 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:14.604118 20635 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:01:14.608596 20635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:42167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:14.609728 20635 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:14.610617 20635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:14.617547 20641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:14.617671 20640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:14.617725 20643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:14.618047 20635 server_base.cc:1061] running on GCE node
I20260430 08:01:14.618474 20635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:14.619118 20635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:14.620357 20635 hybrid_clock.cc:648] HybridClock initialized: now 1777536074620329 us; error 47 us; skew 500 ppm
Apr 30 08:01:14 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536074, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:14.623371 20635 init.cc:377] Logged in from keytab as oryx/127.13.94.130@KRBTEST.COM (short username oryx)
I20260430 08:01:14.624370 20635 webserver.cc:492] Webserver started at http://127.13.94.130:33747/ using document root <none> and password file <none>
I20260430 08:01:14.624980 20635 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:14.625028 20635 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:14.625181 20635 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:14.626829 20635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "42439d79f4494fef9d7096a3e096a023"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "dc138d8c5db9290c8904e8ddb9640127"
server_key_iv: "3c8b6e3172d5ad1e985669c943d871a2"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.627255 20635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "42439d79f4494fef9d7096a3e096a023"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "dc138d8c5db9290c8904e8ddb9640127"
server_key_iv: "3c8b6e3172d5ad1e985669c943d871a2"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.630821 20635 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:01:14.633279 20650 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.634323 20635 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:14.634454 20635 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "42439d79f4494fef9d7096a3e096a023"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "dc138d8c5db9290c8904e8ddb9640127"
server_key_iv: "3c8b6e3172d5ad1e985669c943d871a2"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.634558 20635 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:14.653504 20635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:14.656551 20635 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:14.656947 20635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:14.657588 20635 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:14.658495 20635 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:14.658543 20635 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.658605 20635 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:14.658638 20635 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.667716 20635 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:34387
I20260430 08:01:14.667737 20763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:34387 every 8 connection(s)
I20260430 08:01:14.668816 20635 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:01:14.672998 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 20635
I20260430 08:01:14.673089 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:01:14.673322 13690 external_mini_cluster.cc:1468] Setting key f639a7a677930326a32ec2f7934e2b0d
Apr 30 08:01:14 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536074, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:14.682555 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:14.670602 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:59909 (local address 127.13.94.190:42167)
0430 08:01:14.670786 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:14.670791 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:14.671479 (+ 688us) server_negotiation.cc:408] Connection header received
0430 08:01:14.672266 (+ 787us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:14.672269 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:14.672319 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:14.672398 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:14.673808 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.674535 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:14.675519 (+ 984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.676191 (+ 672us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:14.679060 (+ 2869us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:14.679085 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:14.679087 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:14.679113 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:14.680614 (+ 1501us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:14.681151 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:14.681155 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:14.681157 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:14.681212 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:14.681504 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:14.681508 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:14.681509 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:14.681666 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:14.681772 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:14.682255 (+ 483us) server_negotiation.cc:300] Negotiation successful
0430 08:01:14.682392 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for oryx/127.13.94.131@KRBTEST.COM; defaulting to no policy
I20260430 08:01:14.683197 20766 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:14.670936 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.130:59909)
0430 08:01:14.671344 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:14.671376 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:14.672083 (+ 707us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:14.672547 (+ 464us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:14.672555 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:14.673022 (+ 467us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:14.673575 (+ 553us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:14.673588 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.674700 (+ 1112us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:14.674704 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:14.675342 (+ 638us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:14.675353 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.675559 (+ 206us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:14.677227 (+ 1668us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:14.677254 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:14.678899 (+ 1645us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:14.680783 (+ 1884us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:14.680790 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:14.680804 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:14.681044 (+ 240us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:14.681321 (+ 277us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:14.681324 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:14.681325 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:14.681413 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:14.681773 (+ 360us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:14.681780 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:14.682008 (+ 228us) client_negotiation.cc:770] Sending connection context
0430 08:01:14.682208 (+ 200us) client_negotiation.cc:241] Negotiation successful
0430 08:01:14.682460 (+ 252us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260430 08:01:14.684365 20764 heartbeater.cc:344] Connected to a master server at 127.13.94.190:42167
I20260430 08:01:14.684620 20764 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:14.685153 20764 heartbeater.cc:507] Master 127.13.94.190:42167 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.131@KRBTEST.COM" created.
I20260430 08:01:14.686287 20435 ts_manager.cc:194] Registered new tserver with Master: 42439d79f4494fef9d7096a3e096a023 (127.13.94.130:34387)
I20260430 08:01:14.687059 20435 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:59909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:14.728014 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:42167
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:14.830225 20771 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:14.830456 20771 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:14.830513 20771 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:14.833913 20771 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:14.833976 20771 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:14.834098 20771 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:01:14.838413 20771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:42167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:14.839514 20771 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:14.840413 20771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:14.847602 20777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:14.847743 20776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:14.847602 20779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:14.847978 20771 server_base.cc:1061] running on GCE node
I20260430 08:01:14.848404 20771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:14.849058 20771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:14.850214 20771 hybrid_clock.cc:648] HybridClock initialized: now 1777536074850198 us; error 30 us; skew 500 ppm
Apr 30 08:01:14 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536074, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:14.853157 20771 init.cc:377] Logged in from keytab as oryx/127.13.94.131@KRBTEST.COM (short username oryx)
I20260430 08:01:14.854192 20771 webserver.cc:492] Webserver started at http://127.13.94.131:41835/ using document root <none> and password file <none>
I20260430 08:01:14.854751 20771 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:14.854799 20771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:14.855072 20771 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:14.856814 20771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "7183343b3df947c6817050cb5ae11661"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "30deb78fd66573e9ffa87321cd7b9fa8"
server_key_iv: "76e9f10027e476041d03df826ac1732e"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.857292 20771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "7183343b3df947c6817050cb5ae11661"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "30deb78fd66573e9ffa87321cd7b9fa8"
server_key_iv: "76e9f10027e476041d03df826ac1732e"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.860512 20771 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:01:14.862689 20786 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.863637 20771 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:14.863754 20771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "7183343b3df947c6817050cb5ae11661"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "30deb78fd66573e9ffa87321cd7b9fa8"
server_key_iv: "76e9f10027e476041d03df826ac1732e"
server_key_version: "encryptionkey@0"
I20260430 08:01:14.863853 20771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:14.876003 20771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:14.878727 20771 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:14.878947 20771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:14.879496 20771 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:14.880339 20771 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:14.880386 20771 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.880452 20771 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:14.880492 20771 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:14.890163 20771 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:33273
I20260430 08:01:14.890178 20899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:33273 every 8 connection(s)
I20260430 08:01:14.891069 20771 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:01:14.893970 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 20771
I20260430 08:01:14.894093 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:01:14.894345 13690 external_mini_cluster.cc:1468] Setting key 1af49da5fc4f59c3d582590be751b582
Apr 30 08:01:14 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536074, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:14.903569 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:14.892780 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:59819 (local address 127.13.94.190:42167)
0430 08:01:14.892953 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:14.892957 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:14.893632 (+ 675us) server_negotiation.cc:408] Connection header received
0430 08:01:14.894735 (+ 1103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:14.894739 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:14.894806 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:14.894894 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:14.896294 (+ 1400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.896995 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:14.897585 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.897722 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:14.900471 (+ 2749us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:14.900494 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:14.900497 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:14.900520 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:14.901757 (+ 1237us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:14.902238 (+ 481us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:14.902242 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:14.902244 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:14.902294 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:14.902581 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:14.902585 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:14.902587 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:14.902734 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:14.902835 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:14.903294 (+ 459us) server_negotiation.cc:300] Negotiation successful
0430 08:01:14.903408 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260430 08:01:14.904168 20902 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:14.893103 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.131:59819)
0430 08:01:14.893486 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:14.893518 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:14.894450 (+ 932us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:14.895074 (+ 624us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:14.895082 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:14.895610 (+ 528us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:14.896150 (+ 540us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:14.896162 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.897104 (+ 942us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:14.897107 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:14.897492 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:14.897498 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:14.897665 (+ 167us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:14.898709 (+ 1044us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:14.898727 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:14.900319 (+ 1592us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:14.901877 (+ 1558us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:14.901883 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:14.901897 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:14.902134 (+ 237us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:14.902404 (+ 270us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:14.902407 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:14.902409 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:14.902490 (+ 81us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:14.902847 (+ 357us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:14.902854 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:14.903063 (+ 209us) client_negotiation.cc:770] Sending connection context
0430 08:01:14.903261 (+ 198us) client_negotiation.cc:241] Negotiation successful
0430 08:01:14.903481 (+ 220us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":91,"threads_started":1}
I20260430 08:01:14.905390 20900 heartbeater.cc:344] Connected to a master server at 127.13.94.190:42167
I20260430 08:01:14.905618 20900 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:14.906091 20900 heartbeater.cc:507] Master 127.13.94.190:42167 requested a full tablet report, sending...
I20260430 08:01:14.907058 20435 ts_manager.cc:194] Registered new tserver with Master: 7183343b3df947c6817050cb5ae11661 (127.13.94.131:33273)
I20260430 08:01:14.907656 20435 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:59819
I20260430 08:01:14.908543 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.94.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.94.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:14.963685 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.189:34257
--webserver_interface=127.13.94.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:42167,127.13.94.189:34257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:15.072392 20907 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:15.072678 20907 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:15.072817 20907 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:15.076260 20907 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:15.076336 20907 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:15.076363 20907 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:15.076385 20907 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:15.076406 20907 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:15.081097 20907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:42167,127.13.94.189:34257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.189:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.94.189
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
W20260430 08:01:15.081305 20907 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.94.190:42167,127.13.94.189:34257'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260430 08:01:15.082222 20907 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:15.083086 20907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:15.088876 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:15.088912 20912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:15.089104 20907 server_base.cc:1061] running on GCE node
W20260430 08:01:15.088943 20913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:15.089768 20907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:15.090705 20907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:15.091900 20907 hybrid_clock.cc:648] HybridClock initialized: now 1777536075091873 us; error 38 us; skew 500 ppm
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:15.094974 20907 init.cc:377] Logged in from keytab as oryx/127.13.94.189@KRBTEST.COM (short username oryx)
I20260430 08:01:15.096060 20907 webserver.cc:492] Webserver started at http://127.13.94.189:33719/ using document root <none> and password file <none>
I20260430 08:01:15.096647 20907 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:15.096747 20907 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:15.096975 20907 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:15.098631 20907 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data/instance:
uuid: "e8c26e5434bf49f7afca8e447329f514"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "d214af0fc09265f350d61cc51cc33b3c"
server_key_iv: "c19d911b5ca1ab6e0b8c8cb4583bc70d"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.099169 20907 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal/instance:
uuid: "e8c26e5434bf49f7afca8e447329f514"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "d214af0fc09265f350d61cc51cc33b3c"
server_key_iv: "c19d911b5ca1ab6e0b8c8cb4583bc70d"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.102733 20907 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:01:15.105195 20922 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:15.106293 20907 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:15.106451 20907 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal
uuid: "e8c26e5434bf49f7afca8e447329f514"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "d214af0fc09265f350d61cc51cc33b3c"
server_key_iv: "c19d911b5ca1ab6e0b8c8cb4583bc70d"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.106575 20907 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:15.120494 20907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:15.123342 20907 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:15.123551 20907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:15.131039 20907 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.189:34257
I20260430 08:01:15.131047 20974 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.189:34257 every 8 connection(s)
I20260430 08:01:15.132046 20907 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data/info.pb
I20260430 08:01:15.134841 20975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:15.138463 20975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } has no permanent_uuid. Determining permanent_uuid...
I20260430 08:01:15.140131 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 20907
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.189@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.94.189@KRBTEST.COM
I20260430 08:01:15.152642 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.140399 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.13.94.190:42167)
0430 08:01:15.140559 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.140563 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.141446 (+ 883us) server_negotiation.cc:408] Connection header received
0430 08:01:15.142325 (+ 879us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.142329 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.142382 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.142501 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.143691 (+ 1190us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.144491 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.145284 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.145457 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.147818 (+ 2361us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.147844 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.147846 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.147870 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.150034 (+ 2164us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.150851 (+ 817us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.150856 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.150858 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.150921 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.151305 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.151309 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.151311 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.151570 (+ 259us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.151656 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.152356 (+ 700us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.152470 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 08:01:15.154038 20977 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.140807 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42398)
0430 08:01:15.141300 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.141345 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.142136 (+ 791us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.142576 (+ 440us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.142581 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.143114 (+ 533us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.143508 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.143525 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.144601 (+ 1076us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.144604 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.145174 (+ 570us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.145182 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.145339 (+ 157us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.145921 (+ 582us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.145938 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.147683 (+ 1745us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.150225 (+ 2542us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.150233 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.150242 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.150733 (+ 491us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.151021 (+ 288us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.151026 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.151030 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.151170 (+ 140us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.151747 (+ 577us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.151752 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.152030 (+ 278us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.153003 (+ 973us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.153232 (+ 229us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":141,"threads_started":1}
I20260430 08:01:15.154119 20979 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.141071 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.13.94.189:34257)
0430 08:01:15.141418 (+ 347us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.141423 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.141530 (+ 107us) server_negotiation.cc:408] Connection header received
0430 08:01:15.142156 (+ 626us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.142178 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.142400 (+ 222us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.142575 (+ 175us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.143658 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.144738 (+ 1080us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.146155 (+ 1417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.146665 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.149580 (+ 2915us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.149603 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.149617 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.149649 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.152171 (+ 2522us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.152633 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.152640 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.152641 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.152720 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.152970 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.152974 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.152976 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.153201 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.153421 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.153683 (+ 262us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.153830 (+ 147us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20260430 08:01:15.156306 20975 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } has no permanent_uuid. Determining permanent_uuid...
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.189@KRBTEST.COM for oryx/127.13.94.189@KRBTEST.COM
I20260430 08:01:15.163967 20977 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.156872 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.0.0.1:57670)
0430 08:01:15.157032 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.157045 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.157110 (+ 65us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.157392 (+ 282us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.157395 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.157570 (+ 175us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.157816 (+ 246us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.157823 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.158483 (+ 660us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.158487 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.159024 (+ 537us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.159030 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.159132 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.159625 (+ 493us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.159639 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.160940 (+ 1301us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.162724 (+ 1784us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.162727 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.162729 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.162934 (+ 205us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.163161 (+ 227us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.163163 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.163164 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.163206 (+ 42us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.163569 (+ 363us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.163572 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.163641 (+ 69us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.163754 (+ 113us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.163845 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260430 08:01:15.164048 20979 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.156967 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.13.94.189:34257)
0430 08:01:15.157087 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.157097 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.157110 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:15.157230 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.157233 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.157283 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.157356 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.157929 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.158389 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.159140 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.159318 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.161046 (+ 1728us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.161060 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.161062 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.161083 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.162566 (+ 1483us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.163037 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.163039 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.163041 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.163079 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.163314 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.163317 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.163318 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.163454 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.163545 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.163811 (+ 266us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.163918 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260430 08:01:15.166793 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: Bootstrap starting.
I20260430 08:01:15.169363 20975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:15.170184 20975 log.cc:826] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:15.171942 20975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: No bootstrap required, opened a new log
I20260430 08:01:15.174513 20975 raft_consensus.cc:359] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } }
I20260430 08:01:15.174718 20975 raft_consensus.cc:385] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:15.174816 20975 raft_consensus.cc:740] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c26e5434bf49f7afca8e447329f514, State: Initialized, Role: FOLLOWER
I20260430 08:01:15.175223 20975 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } }
I20260430 08:01:15.175663 20981 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } } }
I20260430 08:01:15.175767 20981 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 08:01:15.176283 20975 sys_catalog.cc:565] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: configured and running, proceeding with master startup.
W20260430 08:01:15.180765 20992 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 08:01:15.180863 20992 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 08:01:15.182904 20907 master_runner.cc:428] Detected that this master e8c26e5434bf49f7afca8e447329f514 is joining an existing cluster
I20260430 08:01:15.182977 20907 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.94.189:34257
I20260430 08:01:15.192461 20979 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.185344 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.13.94.189:34257)
0430 08:01:15.185437 (+ 93us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.185441 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.185454 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:15.185584 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.185587 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.185655 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.185734 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.186950 (+ 1216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.187614 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.188494 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.188968 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.189480 (+ 512us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.189493 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.189495 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.189515 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.190958 (+ 1443us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.191399 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.191402 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.191403 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.191439 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.191676 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.191679 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.191682 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.191814 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.191888 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.192167 (+ 279us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.192321 (+ 154us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":39}
I20260430 08:01:15.192481 20999 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.184977 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.0.0.1:57674)
0430 08:01:15.185348 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.185363 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.185454 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.185768 (+ 314us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.185773 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.186515 (+ 742us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.186812 (+ 297us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.186820 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.187740 (+ 920us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.187771 (+ 31us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.188375 (+ 604us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.188384 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.188498 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.188959 (+ 461us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.188976 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.189347 (+ 371us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.191069 (+ 1722us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.191073 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.191076 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.191308 (+ 232us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.191527 (+ 219us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.191530 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.191533 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.191589 (+ 56us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.191915 (+ 326us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.191918 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.192000 (+ 82us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.192123 (+ 123us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.192251 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":107,"threads_started":1}
I20260430 08:01:15.213865 20998 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.185089 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42408)
0430 08:01:15.185411 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.185426 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.185520 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.185847 (+ 327us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.185850 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.186071 (+ 221us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.186290 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.186295 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.187105 (+ 810us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.187109 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.187652 (+ 543us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.187661 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.187829 (+ 168us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.188465 (+ 636us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.188486 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.209417 (+ 20931us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.212149 (+ 2732us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.212154 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.212157 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.212433 (+ 276us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.212745 (+ 312us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.212749 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.212751 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.212811 (+ 60us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.213281 (+ 470us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.213284 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.213375 (+ 91us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.213521 (+ 146us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.213665 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260430 08:01:15.214221 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.185020 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.13.94.190:42167)
0430 08:01:15.185173 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.185179 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.185497 (+ 318us) server_negotiation.cc:408] Connection header received
0430 08:01:15.185684 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.185691 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.185747 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.185820 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.186416 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.186970 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.187815 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.188051 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.209622 (+ 21571us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.209652 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.209657 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.209712 (+ 55us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.212015 (+ 2303us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.212538 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.212542 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.212545 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.212602 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.212925 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.212929 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.212931 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.213142 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.213212 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.213610 (+ 398us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.213766 (+ 156us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for oryx/127.13.94.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.188@KRBTEST.COM" created.
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536074, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.190@KRBTEST.COM for oryx/127.13.94.189@KRBTEST.COM
I20260430 08:01:15.233599 20979 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.219926 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.13.94.189:34257)
0430 08:01:15.220098 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.220102 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.220118 (+ 16us) server_negotiation.cc:408] Connection header received
0430 08:01:15.220194 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.220199 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.220254 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.220343 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.221484 (+ 1141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.222295 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.223318 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.223552 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.228869 (+ 5317us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.228895 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.228900 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.228948 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.231074 (+ 2126us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.231795 (+ 721us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.231802 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.231806 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.231866 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.232209 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.232213 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.232217 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.232431 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.232512 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.233223 (+ 711us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.233371 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260430 08:01:15.234491 21001 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.219181 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.0.0.1:57684)
0430 08:01:15.219536 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.219573 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.219702 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.220420 (+ 718us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.220427 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.221023 (+ 596us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.221334 (+ 311us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.221346 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.222614 (+ 1268us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.222641 (+ 27us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.223174 (+ 533us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.223184 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.223694 (+ 510us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.224266 (+ 572us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.224283 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.228615 (+ 4332us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.231250 (+ 2635us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.231262 (+ 12us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.231269 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.231649 (+ 380us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.232014 (+ 365us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.232019 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.232022 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.232092 (+ 70us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.232616 (+ 524us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.232625 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.232923 (+ 298us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.234045 (+ 1122us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.234241 (+ 196us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260430 08:01:15.236241 20433 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.13.94.189:34257: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e8c26e5434bf49f7afca8e447329f514"
member_type: NON_VOTER
last_known_addr {
host: "127.13.94.189"
port: 34257
}
attrs {
promote: true
}
}
dest_uuid: "15f647c7facb44c89330f7899a1f3c99"
cas_config_opid_index: -1
I20260430 08:01:15.237084 20433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } }
I20260430 08:01:15.239223 21004 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e8c26e5434bf49f7afca8e447329f514 (127.13.94.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } } }
I20260430 08:01:15.239945 20949 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:15.241875 21004 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.13.94.189:34257
I20260430 08:01:15.242161 20949 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Refusing update from remote peer 15f647c7facb44c89330f7899a1f3c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260430 08:01:15.242470 21004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } } }
I20260430 08:01:15.242578 20907 master.cc:562] Master@127.13.94.189:34257 shutting down...
I20260430 08:01:15.242694 21004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:15.243729 21004 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
W20260430 08:01:15.247144 20421 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260430 08:01:15.248513 20907 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:15.248842 20907 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:15.248962 20907 tablet_replica.cc:333] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: stopping tablet replica
W20260430 08:01:15.251320 20421 consensus_peers.cc:597] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 -> Peer e8c26e5434bf49f7afca8e447329f514 (127.13.94.189:34257): Couldn't send request to peer e8c26e5434bf49f7afca8e447329f514. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for HTTP/127.13.94.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.188@KRBTEST.COM" created.
I20260430 08:01:15.264135 20907 master.cc:584] Master@127.13.94.189:34257 shutdown complete.
I20260430 08:01:15.264222 20907 master_runner.cc:305] Clearing existing system tablet
I20260430 08:01:15.264453 20907 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:15.264530 20907 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:15.265787 20907 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260430 08:01:15.266664 21011 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:15.266918 20907 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:15.267068 20907 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260430 08:01:15.267104 20907 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal
uuid: "e8c26e5434bf49f7afca8e447329f514"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "d214af0fc09265f350d61cc51cc33b3c"
server_key_iv: "c19d911b5ca1ab6e0b8c8cb4583bc70d"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.269208 20907 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:15.272024 20907 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260430 08:01:15.272135 20907 log.cc:1199] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260430 08:01:15.272404 20907 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: Deleting consensus metadata
I20260430 08:01:15.272608 20907 master_runner.cc:315] Copying system tablet from 127.13.94.190:42167
I20260430 08:01:15.274137 20907 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: tablet copy: Beginning tablet copy session from remote peer at address 127.13.94.190:42167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:15.276921 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.188:43089
--webserver_interface=127.13.94.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:42167,127.13.94.189:34257,127.13.94.188:43089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260430 08:01:15.283833 21017 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.274954 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42420)
0430 08:01:15.275258 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.275274 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.275401 (+ 127us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.275679 (+ 278us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.275683 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.276402 (+ 719us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.276788 (+ 386us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.276798 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.277893 (+ 1095us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.277898 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.278505 (+ 607us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.278514 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.278991 (+ 477us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.279553 (+ 562us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.279571 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.279971 (+ 400us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.282233 (+ 2262us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.282237 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.282240 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.282479 (+ 239us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.282786 (+ 307us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.282790 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.282792 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.282848 (+ 56us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.283280 (+ 432us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.283284 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.283369 (+ 85us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.283477 (+ 108us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.283617 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":120,"threads_started":1}
I20260430 08:01:15.283926 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.274954 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.13.94.190:42167)
0430 08:01:15.275135 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.275139 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.275359 (+ 220us) server_negotiation.cc:408] Connection header received
0430 08:01:15.275500 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.275504 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.275556 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.275629 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.276941 (+ 1312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.277752 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.278645 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.278859 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.280118 (+ 1259us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.280144 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.280152 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.280187 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.282100 (+ 1913us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.282592 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.282599 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.282604 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.282678 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.282951 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.282956 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.282961 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.283174 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.283288 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.283566 (+ 278us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.283711 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 08:01:15.284466 20455 tablet_copy_service.cc:140] P 15f647c7facb44c89330f7899a1f3c99: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e8c26e5434bf49f7afca8e447329f514 ({username='oryx', principal='oryx/127.13.94.189@KRBTEST.COM'} at 127.0.0.1:42420)
I20260430 08:01:15.284634 20455 tablet_copy_service.cc:161] P 15f647c7facb44c89330f7899a1f3c99: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e8c26e5434bf49f7afca8e447329f514 at {username='oryx', principal='oryx/127.13.94.189@KRBTEST.COM'} at 127.0.0.1:42420: session id = e8c26e5434bf49f7afca8e447329f514-00000000000000000000000000000000
I20260430 08:01:15.286347 20455 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 08:01:15.288404 20907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260430 08:01:15.290944 20907 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: tablet copy: Starting download of 0 data blocks...
I20260430 08:01:15.291121 20907 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: tablet copy: Starting download of 1 WAL segments...
I20260430 08:01:15.293361 20907 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 08:01:15.295447 20455 tablet_copy_service.cc:342] P 15f647c7facb44c89330f7899a1f3c99: Request end of tablet copy session e8c26e5434bf49f7afca8e447329f514-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.94.189@KRBTEST.COM'} at 127.0.0.1:42420
I20260430 08:01:15.295588 20455 tablet_copy_service.cc:434] P 15f647c7facb44c89330f7899a1f3c99: ending tablet copy session e8c26e5434bf49f7afca8e447329f514-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e8c26e5434bf49f7afca8e447329f514
W20260430 08:01:15.301242 20907 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260430 08:01:15.303751 20907 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:15.305830 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:15.306035 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:15.306172 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:15.306350 20907 server_base.cc:1061] running on GCE node
I20260430 08:01:15.306619 20907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:15.306958 20907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:15.308117 20907 hybrid_clock.cc:648] HybridClock initialized: now 1777536075308093 us; error 41 us; skew 500 ppm
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:15.309855 20907 init.cc:377] Logged in from keytab as oryx/127.13.94.189@KRBTEST.COM (short username oryx)
I20260430 08:01:15.310393 20907 webserver.cc:492] Webserver started at http://127.13.94.189:45747/ using document root <none> and password file <none>
I20260430 08:01:15.310597 20907 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:15.310643 20907 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:15.312062 20907 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260430 08:01:15.313200 21029 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:15.313483 20907 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260430 08:01:15.313555 20907 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal
uuid: "e8c26e5434bf49f7afca8e447329f514"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "d214af0fc09265f350d61cc51cc33b3c"
server_key_iv: "c19d911b5ca1ab6e0b8c8cb4583bc70d"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.313630 20907 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:15.329412 20907 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:15.331578 20907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:15.336473 20907 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.189:34257
I20260430 08:01:15.336690 21081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.189:34257 every 8 connection(s)
I20260430 08:01:15.337153 20907 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-1/data/info.pb
I20260430 08:01:15.338809 21082 sys_catalog.cc:263] Verifying existing consensus state
I20260430 08:01:15.339860 21082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: Bootstrap starting.
I20260430 08:01:15.350198 21082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 08:01:15.350540 21082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: Bootstrap complete.
I20260430 08:01:15.351267 21082 raft_consensus.cc:359] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } }
I20260430 08:01:15.351409 21082 raft_consensus.cc:740] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e8c26e5434bf49f7afca8e447329f514, State: Initialized, Role: LEARNER
I20260430 08:01:15.351542 21082 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } }
I20260430 08:01:15.351930 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } } }
I20260430 08:01:15.352064 21082 sys_catalog.cc:565] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:15.352106 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: This master's current role is: LEARNER
I20260430 08:01:15.355135 21096 catalog_manager.cc:1269] Loaded cluster ID: 85a9016ecce6489d9d2fa6d395880450
I20260430 08:01:15.355296 21096 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: loading cluster ID for follower catalog manager: success
I20260430 08:01:15.357568 21096 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: acquiring CA information for follower catalog manager: success
I20260430 08:01:15.358870 21096 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260430 08:01:15.398604 21018 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:15.398864 21018 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:15.398972 21018 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:15.402515 21018 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:15.402613 21018 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:15.402655 21018 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:15.402719 21018 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:15.402760 21018 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:15.407264 21018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:42167,127.13.94.189:34257,127.13.94.188:43089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.188:43089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.94.188
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:15.408370 21018 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:15.409337 21018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:15.414665 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:15.414687 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:15.414626 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:15.415012 21018 server_base.cc:1061] running on GCE node
I20260430 08:01:15.415539 21018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:15.416415 21018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:15.417598 21018 hybrid_clock.cc:648] HybridClock initialized: now 1777536075417581 us; error 42 us; skew 500 ppm
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:15.420522 21018 init.cc:377] Logged in from keytab as oryx/127.13.94.188@KRBTEST.COM (short username oryx)
I20260430 08:01:15.421674 21018 webserver.cc:492] Webserver started at http://127.13.94.188:32911/ using document root <none> and password file <none>
I20260430 08:01:15.422248 21018 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:15.422323 21018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:15.422544 21018 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:15.424201 21018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data/instance:
uuid: "4887df80ff3040bb865cde1d4ed64d93"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "a7cf9eb231f87baa92da9f84fb8379c9"
server_key_iv: "d1ba642fc409b127548906ed762dbdb1"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.424674 21018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal/instance:
uuid: "4887df80ff3040bb865cde1d4ed64d93"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "a7cf9eb231f87baa92da9f84fb8379c9"
server_key_iv: "d1ba642fc409b127548906ed762dbdb1"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.427874 21018 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260430 08:01:15.430059 21111 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:15.431116 21018 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:01:15.431257 21018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal
uuid: "4887df80ff3040bb865cde1d4ed64d93"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "a7cf9eb231f87baa92da9f84fb8379c9"
server_key_iv: "d1ba642fc409b127548906ed762dbdb1"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.431360 21018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:15.453140 21018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:15.457068 21018 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:15.457280 21018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:15.460537 20628 heartbeater.cc:499] Master 127.13.94.190:42167 was elected leader, sending a full tablet report...
I20260430 08:01:15.465435 21018 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.188:43089
I20260430 08:01:15.465474 21163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.188:43089 every 8 connection(s)
I20260430 08:01:15.466507 21018 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data/info.pb
I20260430 08:01:15.469532 21164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:15.473330 21164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } has no permanent_uuid. Determining permanent_uuid...
I20260430 08:01:15.474674 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 21018
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.188@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.94.188@KRBTEST.COM
I20260430 08:01:15.491703 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.480858 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.13.94.190:42167)
0430 08:01:15.481035 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.481040 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.481221 (+ 181us) server_negotiation.cc:408] Connection header received
0430 08:01:15.482026 (+ 805us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.482030 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.482081 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.482153 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.483267 (+ 1114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.483788 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.484445 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.484642 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.487263 (+ 2621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.487291 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.487293 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.487328 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.489510 (+ 2182us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.490093 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.490097 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.490099 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.490152 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.490512 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.490515 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.490516 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.490720 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.490822 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.491386 (+ 564us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.491523 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260430 08:01:15.492328 21167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.475697 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42422)
0430 08:01:15.481039 (+ 5342us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.481083 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.481804 (+ 721us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.482234 (+ 430us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.482239 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.482753 (+ 514us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.483110 (+ 357us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.483126 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.483900 (+ 774us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.483903 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.484332 (+ 429us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.484339 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.484529 (+ 190us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.485320 (+ 791us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.485337 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.487119 (+ 1782us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.489646 (+ 2527us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.489657 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.489667 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.489982 (+ 315us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.490273 (+ 291us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.490278 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.490280 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.490407 (+ 127us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.490833 (+ 426us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.490840 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.491119 (+ 279us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.491321 (+ 202us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.491590 (+ 269us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5158,"spinlock_wait_cycles":97024,"thread_start_us":230,"threads_started":1}
I20260430 08:01:15.492336 21166 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.475689 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.13.94.188:43089)
0430 08:01:15.481004 (+ 5315us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.481016 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.481059 (+ 43us) server_negotiation.cc:408] Connection header received
0430 08:01:15.481802 (+ 743us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.481828 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.482070 (+ 242us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.482256 (+ 186us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.483144 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.483895 (+ 751us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.484562 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.484821 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.487812 (+ 2991us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.487835 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.487850 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.487879 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.490132 (+ 2253us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.490620 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.490626 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.490628 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.490672 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.490970 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.490972 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.490973 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.491153 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.491323 (+ 170us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.491533 (+ 210us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.491654 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5158,"thread_start_us":140,"threads_started":1}
I20260430 08:01:15.493801 21164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } has no permanent_uuid. Determining permanent_uuid...
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.188@KRBTEST.COM for oryx/127.13.94.189@KRBTEST.COM
I20260430 08:01:15.501847 21167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.494363 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.0.0.1:57694)
0430 08:01:15.494801 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.494814 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.494879 (+ 65us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.495113 (+ 234us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.495116 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.495263 (+ 147us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.495442 (+ 179us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.495447 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.496449 (+ 1002us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.496451 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.496876 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.496883 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.496974 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.497454 (+ 480us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.497467 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.498652 (+ 1185us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.500410 (+ 1758us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.500413 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.500415 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.500569 (+ 154us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.500940 (+ 371us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.500945 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.500948 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.501003 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.501448 (+ 445us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.501451 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.501520 (+ 69us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.501604 (+ 84us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.501723 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301}
I20260430 08:01:15.502033 21168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.494511 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.13.94.189:34257)
0430 08:01:15.494851 (+ 340us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.494855 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.494868 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:15.494960 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.494963 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.495008 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.495088 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.495584 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.496347 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.496978 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.497121 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.498793 (+ 1672us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.498811 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.498814 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.498839 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.500301 (+ 1462us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.500793 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.500796 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.500798 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.500846 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.501153 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.501157 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.501159 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.501343 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.501453 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.501757 (+ 304us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.501864 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":98,"threads_started":1}
I20260430 08:01:15.502733 21164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } has no permanent_uuid. Determining permanent_uuid...
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.188@KRBTEST.COM for oryx/127.13.94.188@KRBTEST.COM
I20260430 08:01:15.510134 21167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.503243 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.0.0.1:32846)
0430 08:01:15.503363 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.503378 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.503450 (+ 72us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.503787 (+ 337us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.503789 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.503915 (+ 126us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.504063 (+ 148us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.504068 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.504830 (+ 762us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.504833 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.505154 (+ 321us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.505159 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.505246 (+ 87us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.505724 (+ 478us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.505737 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.506863 (+ 1126us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.508728 (+ 1865us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.508731 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.508734 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.508902 (+ 168us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.509140 (+ 238us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.509144 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.509147 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.509195 (+ 48us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.509615 (+ 420us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.509627 (+ 12us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.509702 (+ 75us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.509838 (+ 136us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.509954 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260430 08:01:15.510283 21166 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.503412 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.13.94.188:43089)
0430 08:01:15.503605 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.503609 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.503621 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:01:15.503650 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.503652 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.503687 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.503760 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.504183 (+ 423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.504646 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.505246 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.505391 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.507015 (+ 1624us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.507030 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.507032 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.507052 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.508555 (+ 1503us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.508998 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.509004 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.509008 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.509052 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.509278 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.509283 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.509285 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.509452 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.509559 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.509944 (+ 385us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.510062 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260430 08:01:15.514156 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Bootstrap starting.
I20260430 08:01:15.517124 21164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:15.517788 21164 log.cc:826] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:15.519558 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: No bootstrap required, opened a new log
I20260430 08:01:15.522150 21164 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } }
I20260430 08:01:15.522320 21164 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:15.522353 21164 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4887df80ff3040bb865cde1d4ed64d93, State: Initialized, Role: FOLLOWER
I20260430 08:01:15.522716 21164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } }
I20260430 08:01:15.523123 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } } }
I20260430 08:01:15.523231 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 08:01:15.523793 21164 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: configured and running, proceeding with master startup.
W20260430 08:01:15.528681 21181 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 08:01:15.528800 21181 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 08:01:15.532071 21018 master_runner.cc:428] Detected that this master 4887df80ff3040bb865cde1d4ed64d93 is joining an existing cluster
I20260430 08:01:15.532137 21018 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.94.188:43089
I20260430 08:01:15.550173 21001 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.547082 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42438)
0430 08:01:15.547226 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.547241 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.547346 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.547855 (+ 509us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.547858 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.547880 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:15.548070 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.548075 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.549202 (+ 1127us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.549205 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.549787 (+ 582us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.549792 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.549908 (+ 116us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.549971 (+ 63us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.550013 (+ 42us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.550062 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260430 08:01:15.550717 21190 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.547245 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.13.94.190:42167)
0430 08:01:15.547606 (+ 361us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.547611 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.547626 (+ 15us) server_negotiation.cc:408] Connection header received
0430 08:01:15.547681 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.547685 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.547737 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.547829 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:15.548192 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.549081 (+ 889us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.549929 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.550426 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.550488 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.550535 (+ 47us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.550589 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":139,"threads_started":1}
I20260430 08:01:15.554903 21001 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.552037 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.0.0.1:57720)
0430 08:01:15.552155 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.552167 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.552232 (+ 65us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.552742 (+ 510us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.552748 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.552773 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:15.552930 (+ 157us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.552937 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.553805 (+ 868us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.553810 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.554480 (+ 670us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.554488 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.554621 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.554639 (+ 18us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.554703 (+ 64us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.554764 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260430 08:01:15.555486 21191 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.552153 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.13.94.189:34257)
0430 08:01:15.552440 (+ 287us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.552443 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.552455 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:01:15.552497 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.552499 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.552548 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.552664 (+ 116us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:15.553052 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.553685 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.554607 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.555128 (+ 521us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.555279 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.555327 (+ 48us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.555385 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":107,"threads_started":1}
I20260430 08:01:15.564230 21187 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.534517 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.0.0.1:57706)
0430 08:01:15.534851 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.534867 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.534975 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.536274 (+ 1299us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.536278 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.557058 (+ 20780us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.557391 (+ 333us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.557400 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.558474 (+ 1074us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.558478 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.559072 (+ 594us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.559080 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.559187 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.559830 (+ 643us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.559848 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.560289 (+ 441us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.562615 (+ 2326us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.562618 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.562621 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.562925 (+ 304us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.563229 (+ 304us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.563232 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.563235 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.563286 (+ 51us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.563669 (+ 383us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.563673 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.563750 (+ 77us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.563859 (+ 109us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.564026 (+ 167us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":140,"threads_started":1}
I20260430 08:01:15.564591 21168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.535550 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.13.94.189:34257)
0430 08:01:15.536042 (+ 492us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.536048 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.536061 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:15.536103 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.536105 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.536142 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.536204 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.557551 (+ 21347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.558335 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.559407 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.559602 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.560427 (+ 825us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.560448 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.560451 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.560479 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.562470 (+ 1991us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.563074 (+ 604us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.563078 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.563080 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.563129 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.563376 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.563380 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.563382 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.563547 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.563631 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.564066 (+ 435us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.564176 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
I20260430 08:01:15.564774 21188 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.535011 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.0.0.1:32852)
0430 08:01:15.535285 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.535298 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.535372 (+ 74us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.536284 (+ 912us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.536286 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.557077 (+ 20791us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.557722 (+ 645us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.557730 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.558422 (+ 692us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.558426 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.559010 (+ 584us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.559019 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.559144 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.560228 (+ 1084us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.560245 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.560600 (+ 355us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.563398 (+ 2798us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.563401 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.563403 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.563600 (+ 197us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.563854 (+ 254us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.563857 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.563862 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.563908 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.564261 (+ 353us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.564264 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.564337 (+ 73us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.564441 (+ 104us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.564593 (+ 152us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":100,"threads_started":1}
I20260430 08:01:15.564925 21166 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.535375 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.13.94.188:43089)
0430 08:01:15.535671 (+ 296us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.535673 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.535683 (+ 10us) server_negotiation.cc:408] Connection header received
0430 08:01:15.535733 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.535736 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.535769 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.535869 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.557818 (+ 21949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.558303 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.559263 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.559460 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.560772 (+ 1312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.560789 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.560793 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.560816 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.563294 (+ 2478us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.563706 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.563710 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.563712 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.563758 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.563995 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.564001 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.564005 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.564145 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.564227 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.564631 (+ 404us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.564758 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260430 08:01:15.569496 21189 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.535085 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42430)
0430 08:01:15.535371 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.535385 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.535519 (+ 134us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.535875 (+ 356us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.535878 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.557332 (+ 21454us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.557645 (+ 313us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.557652 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.559597 (+ 1945us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.559601 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.561296 (+ 1695us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.561308 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.561993 (+ 685us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.562949 (+ 956us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.563002 (+ 53us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.565057 (+ 2055us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.567554 (+ 2497us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.567559 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.567561 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.567982 (+ 421us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.568285 (+ 303us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.568288 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.568290 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.568426 (+ 136us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.568904 (+ 478us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.568907 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.568992 (+ 85us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.569151 (+ 159us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.569309 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":75,"threads_started":1}
I20260430 08:01:15.569628 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.534562 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.13.94.190:42167)
0430 08:01:15.534716 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.534720 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.535450 (+ 730us) server_negotiation.cc:408] Connection header received
0430 08:01:15.535659 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.535664 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.535723 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.535808 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.558445 (+ 22637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.559160 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.561434 (+ 2274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.561655 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.565507 (+ 3852us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.565525 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.565528 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.565558 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.567435 (+ 1877us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.568096 (+ 661us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.568102 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.568105 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.568159 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.568515 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.568519 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.568521 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.568756 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.568850 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.569358 (+ 508us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.569476 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536074, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.190@KRBTEST.COM for oryx/127.13.94.188@KRBTEST.COM
I20260430 08:01:15.581099 21166 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.573265 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.13.94.188:43089)
0430 08:01:15.573402 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.573409 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.573423 (+ 14us) server_negotiation.cc:408] Connection header received
0430 08:01:15.573651 (+ 228us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.573654 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.573711 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.573792 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.574479 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.575184 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.575906 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.576110 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.577979 (+ 1869us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.577995 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.577998 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.578022 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.579510 (+ 1488us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.579973 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.579976 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.579977 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.580021 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.580311 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.580314 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.580316 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.580467 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.580565 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.580888 (+ 323us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.580981 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 08:01:15.581110 21001 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.573179 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.0.0.1:32854)
0430 08:01:15.573370 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.573384 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.573507 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.573908 (+ 401us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.573912 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.574115 (+ 203us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.574349 (+ 234us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.574356 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.575278 (+ 922us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.575282 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.575773 (+ 491us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.575781 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.575913 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.576422 (+ 509us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.576438 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.577852 (+ 1414us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.579621 (+ 1769us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.579625 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.579627 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.579853 (+ 226us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.580121 (+ 268us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.580125 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.580127 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.580194 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.580572 (+ 378us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.580576 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.580665 (+ 89us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.580795 (+ 130us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.580926 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260430 08:01:15.582181 20433 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.13.94.188:43089: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93"
member_type: NON_VOTER
last_known_addr {
host: "127.13.94.188"
port: 43089
}
attrs {
promote: true
}
}
dest_uuid: "15f647c7facb44c89330f7899a1f3c99"
cas_config_opid_index: 5
I20260430 08:01:15.582813 20433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } }
I20260430 08:01:15.583921 21005 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4887df80ff3040bb865cde1d4ed64d93 (127.13.94.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.584378 21056 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 LEARNER]: Refusing update from remote peer 15f647c7facb44c89330f7899a1f3c99: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260430 08:01:15.584457 21138 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:15.584830 21004 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260430 08:01:15.585616 21005 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.13.94.188:43089
I20260430 08:01:15.586387 21138 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Refusing update from remote peer 15f647c7facb44c89330f7899a1f3c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260430 08:01:15.586941 21018 master.cc:562] Master@127.13.94.188:43089 shutting down...
I20260430 08:01:15.586709 21056 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4887df80ff3040bb865cde1d4ed64d93 (127.13.94.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.588470 21193 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.588593 21193 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: This master's current role is: LEARNER
I20260430 08:01:15.588929 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15f647c7facb44c89330f7899a1f3c99. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.589073 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: This master's current role is: LEARNER
I20260430 08:01:15.589951 21194 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:15.590385 21005 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: attempting to promote NON_VOTER e8c26e5434bf49f7afca8e447329f514 to VOTER
I20260430 08:01:15.590557 21004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: NON_VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: true } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.590670 21004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:15.591092 21005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } }
W20260430 08:01:15.594193 20421 consensus_peers.cc:597] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 -> Peer 4887df80ff3040bb865cde1d4ed64d93 (127.13.94.188:43089): Couldn't send request to peer 4887df80ff3040bb865cde1d4ed64d93. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260430 08:01:15.594240 21056 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 LEARNER]: Refusing update from remote peer 15f647c7facb44c89330f7899a1f3c99: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260430 08:01:15.594626 21195 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260430 08:01:15.594941 20435 master.cc:680] Remote error: unable to get registration information for peer 4887df80ff3040bb865cde1d4ed64d93: Service unavailable: service kudu.master.MasterService not registered on Master
I20260430 08:01:15.596315 21194 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e8c26e5434bf49f7afca8e447329f514 (127.13.94.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.596930 21056 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e8c26e5434bf49f7afca8e447329f514 (127.13.94.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.597415 21018 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:15.597688 21018 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:15.597797 21018 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: stopping tablet replica
I20260430 08:01:15.598001 21004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.598114 21004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:15.598042 21005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.598284 21005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:15.599431 21085 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.599546 21085 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 08:01:15.612947 21018 master.cc:584] Master@127.13.94.188:43089 shutdown complete.
I20260430 08:01:15.613090 21018 master_runner.cc:305] Clearing existing system tablet
I20260430 08:01:15.613361 21018 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:15.613418 21018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:15.617762 21018 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260430 08:01:15.619015 21197 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:15.619311 21018 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:15.619639 21018 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260430 08:01:15.619743 21018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal
uuid: "4887df80ff3040bb865cde1d4ed64d93"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "a7cf9eb231f87baa92da9f84fb8379c9"
server_key_iv: "d1ba642fc409b127548906ed762dbdb1"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.620446 21001 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.620111 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.0.0.1:32860)
0430 08:01:15.620258 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.620344 (+ 86us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260430 08:01:15.620787 20433 master.cc:680] Network error: unable to get registration information for peer 4887df80ff3040bb865cde1d4ed64d93: Client connection negotiation failed: client connection to 127.13.94.188:43089: connect: Connection refused (error 111)
I20260430 08:01:15.622432 21018 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:15.624315 21018 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260430 08:01:15.624415 21018 log.cc:1199] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260430 08:01:15.624686 21018 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Deleting consensus metadata
I20260430 08:01:15.624881 21018 master_runner.cc:315] Copying system tablet from 127.13.94.190:42167
I20260430 08:01:15.626163 21018 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: tablet copy: Beginning tablet copy session from remote peer at address 127.13.94.190:42167
I20260430 08:01:15.633874 20474 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.626823 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.13.94.190:42167)
0430 08:01:15.626975 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.626979 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.627092 (+ 113us) server_negotiation.cc:408] Connection header received
0430 08:01:15.627237 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.627240 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.627287 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.627361 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:15.628079 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.628566 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.629469 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.629615 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.630664 (+ 1049us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:15.630681 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:15.630683 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:15.630714 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:15.632240 (+ 1526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.632749 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.632754 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.632758 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.632804 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:15.633074 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:15.633076 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:15.633077 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:15.633259 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:15.633337 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.633652 (+ 315us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.633751 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 08:01:15.633966 21203 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.626684 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42440)
0430 08:01:15.626988 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.627006 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.627140 (+ 134us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.627410 (+ 270us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.627414 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.627657 (+ 243us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:15.627951 (+ 294us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.627960 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.628687 (+ 727us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.628692 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.629344 (+ 652us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.629356 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.629494 (+ 138us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.630076 (+ 582us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:15.630096 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:15.630522 (+ 426us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:15.632357 (+ 1835us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.632361 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.632364 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.632600 (+ 236us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.632910 (+ 310us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:15.632914 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:15.632917 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:15.632980 (+ 63us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:15.633382 (+ 402us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:15.633385 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:15.633479 (+ 94us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.633605 (+ 126us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.633744 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":114,"threads_started":1}
I20260430 08:01:15.634428 20455 tablet_copy_service.cc:140] P 15f647c7facb44c89330f7899a1f3c99: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4887df80ff3040bb865cde1d4ed64d93 ({username='oryx', principal='oryx/127.13.94.188@KRBTEST.COM'} at 127.0.0.1:42440)
I20260430 08:01:15.634531 20455 tablet_copy_service.cc:161] P 15f647c7facb44c89330f7899a1f3c99: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4887df80ff3040bb865cde1d4ed64d93 at {username='oryx', principal='oryx/127.13.94.188@KRBTEST.COM'} at 127.0.0.1:42440: session id = 4887df80ff3040bb865cde1d4ed64d93-00000000000000000000000000000000
I20260430 08:01:15.635320 20455 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 08:01:15.636583 21018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260430 08:01:15.638406 21018 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: tablet copy: Starting download of 0 data blocks...
I20260430 08:01:15.638592 21018 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: tablet copy: Starting download of 1 WAL segments...
I20260430 08:01:15.640278 21018 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 08:01:15.641985 20455 tablet_copy_service.cc:342] P 15f647c7facb44c89330f7899a1f3c99: Request end of tablet copy session 4887df80ff3040bb865cde1d4ed64d93-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.94.188@KRBTEST.COM'} at 127.0.0.1:42440
I20260430 08:01:15.642084 20455 tablet_copy_service.cc:434] P 15f647c7facb44c89330f7899a1f3c99: ending tablet copy session 4887df80ff3040bb865cde1d4ed64d93-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4887df80ff3040bb865cde1d4ed64d93
W20260430 08:01:15.646559 21018 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260430 08:01:15.648845 21018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:15.650494 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:15.650570 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:15.650776 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:15.651371 21018 server_base.cc:1061] running on GCE node
I20260430 08:01:15.651546 21018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:15.651865 21018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:15.653003 21018 hybrid_clock.cc:648] HybridClock initialized: now 1777536075652994 us; error 27 us; skew 500 ppm
Apr 30 08:01:15 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536075, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:15.654853 21018 init.cc:377] Logged in from keytab as oryx/127.13.94.188@KRBTEST.COM (short username oryx)
I20260430 08:01:15.655458 21018 webserver.cc:492] Webserver started at http://127.13.94.188:39177/ using document root <none> and password file <none>
I20260430 08:01:15.655663 21018 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:15.655719 21018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:15.657117 21001 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.656779 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.0.0.1:32874)
0430 08:01:15.656964 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.657022 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.188:43089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260430 08:01:15.657146 21018 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
W20260430 08:01:15.657351 20433 master.cc:680] Network error: unable to get registration information for peer 4887df80ff3040bb865cde1d4ed64d93: Client connection negotiation failed: client connection to 127.13.94.188:43089: connect: Connection refused (error 111)
I20260430 08:01:15.658224 21214 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:15.658501 21018 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260430 08:01:15.658584 21018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal
uuid: "4887df80ff3040bb865cde1d4ed64d93"
format_stamp: "Formatted at 2026-04-30 08:01:15 on dist-test-slave-9bxp"
server_key: "a7cf9eb231f87baa92da9f84fb8379c9"
server_key_iv: "d1ba642fc409b127548906ed762dbdb1"
server_key_version: "encryptionkey@0"
I20260430 08:01:15.658671 21018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:15.667666 21018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:15.669888 21018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:15.674623 21018 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.188:43089
I20260430 08:01:15.674984 21266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.188:43089 every 8 connection(s)
I20260430 08:01:15.675310 21018 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/master-2/data/info.pb
I20260430 08:01:15.677191 21267 sys_catalog.cc:263] Verifying existing consensus state
I20260430 08:01:15.678340 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Bootstrap starting.
I20260430 08:01:15.689443 20764 heartbeater.cc:499] Master 127.13.94.190:42167 was elected leader, sending a full tablet report...
I20260430 08:01:15.691493 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 08:01:15.691908 21267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Bootstrap complete.
I20260430 08:01:15.692652 21267 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } }
I20260430 08:01:15.692849 21267 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4887df80ff3040bb865cde1d4ed64d93, State: Initialized, Role: LEARNER
I20260430 08:01:15.693001 21267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } }
I20260430 08:01:15.693589 21267 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:15.693624 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:15.693830 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: This master's current role is: LEARNER
I20260430 08:01:15.697013 21281 catalog_manager.cc:1269] Loaded cluster ID: 85a9016ecce6489d9d2fa6d395880450
I20260430 08:01:15.697069 21281 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: loading cluster ID for follower catalog manager: success
I20260430 08:01:15.698721 21281 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: acquiring CA information for follower catalog manager: success
I20260430 08:01:15.699771 21281 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260430 08:01:15.729108 21001 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.725597 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.0.0.1:32882)
0430 08:01:15.725815 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:15.725833 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:15.725935 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:15.726351 (+ 416us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:15.726354 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:15.726374 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:15.726546 (+ 172us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.726550 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.727944 (+ 1394us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.727947 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:15.728581 (+ 634us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:15.728588 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.728689 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.728780 (+ 91us) client_negotiation.cc:770] Sending connection context
0430 08:01:15.728855 (+ 75us) client_negotiation.cc:241] Negotiation successful
0430 08:01:15.728931 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260430 08:01:15.729799 21282 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:15.725724 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.13.94.188:43089)
0430 08:01:15.726064 (+ 340us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:15.726071 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:15.726090 (+ 19us) server_negotiation.cc:408] Connection header received
0430 08:01:15.726150 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:15.726153 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:15.726217 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:15.726337 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:15.726666 (+ 329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.727803 (+ 1137us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:15.728759 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:15.729371 (+ 612us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:15.729519 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:15.729583 (+ 64us) server_negotiation.cc:300] Negotiation successful
0430 08:01:15.729645 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
I20260430 08:01:15.910122 20900 heartbeater.cc:499] Master 127.13.94.190:42167 was elected leader, sending a full tablet report...
I20260430 08:01:16.019093 21241 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260430 08:01:16.020191 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15f647c7facb44c89330f7899a1f3c99. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:16.020392 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: This master's current role is: LEARNER
I20260430 08:01:16.527606 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: NON_VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: true } } }
I20260430 08:01:16.527817 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:16.528785 21285 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: attempting to promote NON_VOTER 4887df80ff3040bb865cde1d4ed64d93 to VOTER
I20260430 08:01:16.529196 21285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } }
I20260430 08:01:16.530607 21056 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Refusing update from remote peer 15f647c7facb44c89330f7899a1f3c99: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260430 08:01:16.530652 21241 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 LEARNER]: Refusing update from remote peer 15f647c7facb44c89330f7899a1f3c99: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260430 08:01:16.531158 21285 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260430 08:01:16.531412 21283 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260430 08:01:16.533308 21283 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4887df80ff3040bb865cde1d4ed64d93 (127.13.94.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:16.533819 21056 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4887df80ff3040bb865cde1d4ed64d93 (127.13.94.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:16.535104 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:16.535224 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:16.535269 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:16.535346 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:16.535429 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:16.535378 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:16.535262 21241 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4887df80ff3040bb865cde1d4ed64d93 (127.13.94.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:16.535861 21289 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:16.536001 21289 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 08:01:16.536952 21290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15f647c7facb44c89330f7899a1f3c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:16.537040 21290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 08:01:16.654340 21292 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.642889 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.13.94.189:34257)
0430 08:01:16.643216 (+ 327us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.643220 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.643238 (+ 18us) server_negotiation.cc:408] Connection header received
0430 08:01:16.643300 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.643302 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.643344 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.643422 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:16.644296 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.644934 (+ 638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.645614 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.645774 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.650523 (+ 4749us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:16.650546 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:16.650548 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:16.650582 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:16.652245 (+ 1663us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.652891 (+ 646us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.652896 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.652898 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.652957 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.653257 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.653260 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.653261 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.653496 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:16.653629 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.654074 (+ 445us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.654196 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
I20260430 08:01:16.658151 21293 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.655221 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42452)
0430 08:01:16.655473 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:16.655486 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:16.655558 (+ 72us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:16.655865 (+ 307us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:16.655874 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:16.655895 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:16.656087 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.656092 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.657037 (+ 945us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.657041 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:16.657636 (+ 595us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.657642 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.657742 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.657755 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 08:01:16.657792 (+ 37us) client_negotiation.cc:241] Negotiation successful
0430 08:01:16.657927 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
I20260430 08:01:16.658604 21294 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.655348 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.13.94.190:42167)
0430 08:01:16.655634 (+ 286us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.655637 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.655650 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:16.655694 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.655697 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.655740 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.655818 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:16.656224 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.656916 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.657764 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.658229 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.658286 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.658404 (+ 118us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.658489 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":133,"threads_started":1}
I20260430 08:01:16.662392 21293 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.660009 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.0.0.1:57742)
0430 08:01:16.660130 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:16.660145 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:16.660229 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:16.660417 (+ 188us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:16.660420 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:16.660435 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:16.660578 (+ 143us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.660582 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.661552 (+ 970us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.661554 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:16.662112 (+ 558us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.662118 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.662209 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.662221 (+ 12us) client_negotiation.cc:770] Sending connection context
0430 08:01:16.662257 (+ 36us) client_negotiation.cc:241] Negotiation successful
0430 08:01:16.662296 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260430 08:01:16.662861 21292 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.660070 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.13.94.189:34257)
0430 08:01:16.660223 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.660227 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.660240 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:16.660292 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.660294 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.660332 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.660399 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:16.660754 (+ 355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.661434 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.662225 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.662611 (+ 386us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.662654 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.662719 (+ 65us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.662772 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260430 08:01:16.666813 21293 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.664062 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.0.0.1:32888)
0430 08:01:16.664180 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:16.664196 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:16.664267 (+ 71us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:16.664950 (+ 683us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:16.664953 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:16.664976 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:16.665120 (+ 144us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.665124 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.665853 (+ 729us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.665858 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:16.666425 (+ 567us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.666431 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.666561 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.666576 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 08:01:16.666627 (+ 51us) client_negotiation.cc:241] Negotiation successful
0430 08:01:16.666679 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260430 08:01:16.667258 21295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.664352 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.13.94.188:43089)
0430 08:01:16.664625 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.664629 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.664640 (+ 11us) server_negotiation.cc:408] Connection header received
0430 08:01:16.664747 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.664751 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.664797 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.664897 (+ 100us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:16.665235 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.665742 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.666547 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.667025 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.667057 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.667106 (+ 49us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.667160 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
I20260430 08:01:16.676496 21295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.669283 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.13.94.188:43089)
0430 08:01:16.669402 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.669406 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.669441 (+ 35us) server_negotiation.cc:408] Connection header received
0430 08:01:16.669559 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.669562 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.669609 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.669691 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:16.670462 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.670942 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.671641 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.671760 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.673039 (+ 1279us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:16.673053 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:16.673055 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:16.673077 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:16.674935 (+ 1858us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.675361 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.675364 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.675366 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.675414 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.675683 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.675686 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.675687 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.675846 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:16.675975 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.676224 (+ 249us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.676348 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260430 08:01:16.680629 21296 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.677342 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42468)
0430 08:01:16.677620 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:16.677635 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:16.677893 (+ 258us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:16.678244 (+ 351us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:16.678247 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:16.678271 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:16.678532 (+ 261us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.678538 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.679303 (+ 765us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.679307 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:16.680235 (+ 928us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.680247 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.680369 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.680386 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:01:16.680434 (+ 48us) client_negotiation.cc:241] Negotiation successful
0430 08:01:16.680491 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":118,"threads_started":1}
I20260430 08:01:16.681537 21294 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.677492 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42468 (local address 127.13.94.190:42167)
0430 08:01:16.677651 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.677654 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.677797 (+ 143us) server_negotiation.cc:408] Connection header received
0430 08:01:16.678057 (+ 260us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.678060 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.678137 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.678209 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:16.678644 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.679198 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.680387 (+ 1189us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.681192 (+ 805us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.681234 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.681312 (+ 78us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.681382 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260430 08:01:16.686081 21296 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.683126 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.0.0.1:57748)
0430 08:01:16.683308 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:16.683324 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:16.683406 (+ 82us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:16.683751 (+ 345us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:16.683755 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:16.683789 (+ 34us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:16.683993 (+ 204us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.683998 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.685103 (+ 1105us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.685109 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:16.685686 (+ 577us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.685692 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.685839 (+ 147us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.685856 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:01:16.685909 (+ 53us) client_negotiation.cc:241] Negotiation successful
0430 08:01:16.685962 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260430 08:01:16.686517 21292 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.683356 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.13.94.189:34257)
0430 08:01:16.683535 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.683539 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.683552 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:16.683601 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.683605 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.683650 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.683744 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:16.684124 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.684976 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.685812 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.686289 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.686324 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.686373 (+ 49us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.686428 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260430 08:01:16.690068 21296 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.687736 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.0.0.1:32908)
0430 08:01:16.687888 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:16.687920 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:16.688006 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:16.688248 (+ 242us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:16.688251 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:16.688265 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:16.688434 (+ 169us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.688439 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.689151 (+ 712us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.689155 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:16.689700 (+ 545us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.689705 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.689828 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.689844 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 08:01:16.689896 (+ 52us) client_negotiation.cc:241] Negotiation successful
0430 08:01:16.689945 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260430 08:01:16.690464 21295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.687828 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.13.94.188:43089)
0430 08:01:16.687985 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.687989 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.688002 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:16.688105 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.688108 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.688151 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.688244 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:16.688531 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.689034 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.689810 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.690261 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.690295 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.690341 (+ 46us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.690388 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"spinlock_wait_cycles":13184}
I20260430 08:01:16.691952 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 20499
I20260430 08:01:16.699433 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:36283
--local_ip_for_outbound_sockets=127.13.94.129
--tserver_master_addrs=127.13.94.190:42167
--webserver_port=40865
--webserver_interface=127.13.94.129
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.94.190:42167,127.13.94.189:34257
--tserver_master_addrs=127.13.94.190:42167,127.13.94.189:34257,127.13.94.188:43089 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:16.809527 21297 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:16.809798 21297 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:16.809877 21297 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:16.813297 21297 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:16.813369 21297 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:16.813452 21297 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:01:16.818835 21297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:36283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=40865
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:42167,127.13.94.189:34257,127.13.94.188:43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:16.820001 21297 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:16.820935 21297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:16.828218 21305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:16.828219 21303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:16.828218 21302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:16.828634 21297 server_base.cc:1061] running on GCE node
I20260430 08:01:16.829066 21297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:16.829681 21297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:16.830842 21297 hybrid_clock.cc:648] HybridClock initialized: now 1777536076830819 us; error 41 us; skew 500 ppm
Apr 30 08:01:16 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536076, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:16.833963 21297 init.cc:377] Logged in from keytab as oryx/127.13.94.129@KRBTEST.COM (short username oryx)
I20260430 08:01:16.835240 21297 webserver.cc:492] Webserver started at http://127.13.94.129:40865/ using document root <none> and password file <none>
I20260430 08:01:16.835801 21297 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:16.835860 21297 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:16.839579 21297 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:16.841588 21312 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:16.842713 21297 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:16.842855 21297 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "fb3b88c84b80445f96a8debdb48b5fdb"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "5254d35c2688935e11458787ccd7773a"
server_key_iv: "99220912bad0f1b99e4aa4f5306961f7"
server_key_version: "encryptionkey@0"
I20260430 08:01:16.843325 21297 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:16.853296 21297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:16.856325 21297 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:16.856537 21297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:16.857261 21297 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:16.858278 21297 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:16.858327 21297 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:16.858399 21297 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:16.858429 21297 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:16.868530 21297 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:36283
I20260430 08:01:16.868542 21425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:36283 every 8 connection(s)
I20260430 08:01:16.869673 21297 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:01:16.875797 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 21297
I20260430 08:01:16.875998 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 20635
Apr 30 08:01:16 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536076, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.188@KRBTEST.COM
Apr 30 08:01:16 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536076, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.189@KRBTEST.COM
Apr 30 08:01:16 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536076, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:16.883760 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:34387
--local_ip_for_outbound_sockets=127.13.94.130
--tserver_master_addrs=127.13.94.190:42167
--webserver_port=33747
--webserver_interface=127.13.94.130
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.94.190:42167,127.13.94.189:34257
--tserver_master_addrs=127.13.94.190:42167,127.13.94.189:34257,127.13.94.188:43089 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260430 08:01:16.886008 21295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.871481 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:39737 (local address 127.13.94.188:43089)
0430 08:01:16.871709 (+ 228us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.871714 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.872360 (+ 646us) server_negotiation.cc:408] Connection header received
0430 08:01:16.873577 (+ 1217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.873582 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.873641 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.873720 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:16.875172 (+ 1452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.875896 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.877542 (+ 1646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.877736 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.881303 (+ 3567us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:16.881329 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:16.881333 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:16.881370 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:16.883320 (+ 1950us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.883962 (+ 642us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.883966 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.883968 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.884017 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.884675 (+ 658us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.884678 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.884680 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.884896 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:16.884971 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.885706 (+ 735us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.885837 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260430 08:01:16.886008 21294 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.871552 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:39357 (local address 127.13.94.190:42167)
0430 08:01:16.871709 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.871714 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.872361 (+ 647us) server_negotiation.cc:408] Connection header received
0430 08:01:16.873461 (+ 1100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.873465 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.873533 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.873613 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:16.875043 (+ 1430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.875727 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.879028 (+ 3301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.879492 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.881611 (+ 2119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:16.881641 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:16.881644 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:16.881672 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:16.883787 (+ 2115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.884251 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.884257 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.884261 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.884316 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.884675 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.884678 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.884680 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.884902 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:16.884971 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.885706 (+ 735us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.885837 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 08:01:16.886556 21292 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.872889 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:35721 (local address 127.13.94.189:34257)
0430 08:01:16.873039 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:16.873043 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:16.873167 (+ 124us) server_negotiation.cc:408] Connection header received
0430 08:01:16.873763 (+ 596us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:16.873766 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:16.873828 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:16.873923 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:16.875014 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.875762 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.877178 (+ 1416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.877372 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.881336 (+ 3964us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:16.881359 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:16.881362 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:16.881394 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:16.884006 (+ 2612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.884406 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.884413 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.884418 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.884475 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:16.884808 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:16.884815 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:16.884820 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:16.885036 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:16.885113 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:16.885718 (+ 605us) server_negotiation.cc:300] Negotiation successful
0430 08:01:16.885858 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 08:01:16.887553 21432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.872783 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.13.94.129:35721)
0430 08:01:16.873063 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:16.873076 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:16.873261 (+ 185us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:16.873948 (+ 687us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:16.873951 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:16.874344 (+ 393us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:16.874859 (+ 515us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.874871 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.875896 (+ 1025us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.875901 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:16.877042 (+ 1141us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.877049 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.877272 (+ 223us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.879282 (+ 2010us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:16.879302 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:16.881098 (+ 1796us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:16.884117 (+ 3019us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:16.884120 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:16.884122 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:16.884320 (+ 198us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:16.884587 (+ 267us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:16.884594 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:16.884599 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:16.884663 (+ 64us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:16.885194 (+ 531us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:16.885199 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:16.885394 (+ 195us) client_negotiation.cc:770] Sending connection context
0430 08:01:16.886882 (+ 1488us) client_negotiation.cc:241] Negotiation successful
0430 08:01:16.887042 (+ 160us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"spinlock_wait_cycles":70400,"thread_start_us":96,"threads_started":1}
I20260430 08:01:16.887562 21430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.871794 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.13.94.129:39737)
0430 08:01:16.872227 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:16.872269 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:16.873254 (+ 985us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:16.873774 (+ 520us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:16.873786 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:16.874800 (+ 1014us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:16.875057 (+ 257us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.875064 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.876895 (+ 1831us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.876900 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:16.877411 (+ 511us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.877422 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.877549 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.878382 (+ 833us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:16.878415 (+ 33us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:16.881074 (+ 2659us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:16.883473 (+ 2399us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:16.883480 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:16.883496 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:16.883842 (+ 346us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:16.884442 (+ 600us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:16.884444 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:16.884446 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:16.884566 (+ 120us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:16.885060 (+ 494us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:16.885066 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:16.885385 (+ 319us) client_negotiation.cc:770] Sending connection context
0430 08:01:16.886260 (+ 875us) client_negotiation.cc:241] Negotiation successful
0430 08:01:16.886625 (+ 365us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":135,"spinlock_wait_cycles":19968,"thread_start_us":121,"threads_started":1}
I20260430 08:01:16.887539 21431 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:16.871794 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.129:39357)
0430 08:01:16.872227 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:16.872269 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:16.873263 (+ 994us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:16.873774 (+ 511us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:16.873786 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:16.874359 (+ 573us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:16.874879 (+ 520us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.874889 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.875893 (+ 1004us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:16.875897 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:16.878864 (+ 2967us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:16.878875 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:16.879005 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:16.879863 (+ 858us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:16.879882 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:16.881448 (+ 1566us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:16.883939 (+ 2491us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:16.883944 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:16.883946 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:16.884140 (+ 194us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:16.884431 (+ 291us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:16.884435 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:16.884437 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:16.884574 (+ 137us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:16.885060 (+ 486us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:16.885066 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:16.885385 (+ 319us) client_negotiation.cc:770] Sending connection context
0430 08:01:16.886260 (+ 875us) client_negotiation.cc:241] Negotiation successful
0430 08:01:16.886569 (+ 309us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":21,"spinlock_wait_cycles":42752,"thread_start_us":122,"threads_started":1}
I20260430 08:01:16.889256 21428 heartbeater.cc:344] Connected to a master server at 127.13.94.189:34257
I20260430 08:01:16.889331 21426 heartbeater.cc:344] Connected to a master server at 127.13.94.188:43089
I20260430 08:01:16.889521 21428 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:16.889521 21426 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:16.890210 21428 heartbeater.cc:507] Master 127.13.94.189:34257 requested a full tablet report, sending...
I20260430 08:01:16.890210 21426 heartbeater.cc:507] Master 127.13.94.188:43089 requested a full tablet report, sending...
I20260430 08:01:16.890833 21427 heartbeater.cc:344] Connected to a master server at 127.13.94.190:42167
I20260430 08:01:16.890928 21427 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:16.891134 21427 heartbeater.cc:507] Master 127.13.94.190:42167 requested a full tablet report, sending...
I20260430 08:01:16.891961 21046 ts_manager.cc:194] Registered new tserver with Master: fb3b88c84b80445f96a8debdb48b5fdb (127.13.94.129:36283)
I20260430 08:01:16.891964 21231 ts_manager.cc:194] Registered new tserver with Master: fb3b88c84b80445f96a8debdb48b5fdb (127.13.94.129:36283)
I20260430 08:01:16.893132 20433 ts_manager.cc:194] Re-registered known tserver with Master: fb3b88c84b80445f96a8debdb48b5fdb (127.13.94.129:36283)
I20260430 08:01:16.894066 20433 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:39357
W20260430 08:01:16.997304 21433 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:16.997599 21433 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:16.997695 21433 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:17.001389 21433 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:17.001504 21433 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:17.001626 21433 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:01:17.006475 21433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:34387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=33747
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:42167,127.13.94.189:34257,127.13.94.188:43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:17.007699 21433 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:17.008626 21433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260430 08:01:17.015646 21433 server_base.cc:1061] running on GCE node
W20260430 08:01:17.015542 21438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:17.015592 21439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:17.015542 21441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:17.016379 21433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:17.017040 21433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:17.018220 21433 hybrid_clock.cc:648] HybridClock initialized: now 1777536077018194 us; error 43 us; skew 500 ppm
Apr 30 08:01:17 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536077, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:17.021672 21433 init.cc:377] Logged in from keytab as oryx/127.13.94.130@KRBTEST.COM (short username oryx)
I20260430 08:01:17.022893 21433 webserver.cc:492] Webserver started at http://127.13.94.130:33747/ using document root <none> and password file <none>
I20260430 08:01:17.023494 21433 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:17.023586 21433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:17.027237 21433 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:01:17.029340 21448 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:17.030570 21433 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:17.030704 21433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "42439d79f4494fef9d7096a3e096a023"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "dc138d8c5db9290c8904e8ddb9640127"
server_key_iv: "3c8b6e3172d5ad1e985669c943d871a2"
server_key_version: "encryptionkey@0"
I20260430 08:01:17.031127 21433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:17.054385 21433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:17.057557 21433 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:17.057768 21433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:17.058404 21433 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:17.059360 21433 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:17.059432 21433 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:17.059502 21433 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:17.059551 21433 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:17.069490 21433 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:34387
I20260430 08:01:17.069473 21561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:34387 every 8 connection(s)
I20260430 08:01:17.070559 21433 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:01:17.071138 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 21433
I20260430 08:01:17.071331 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 20771
I20260430 08:01:17.079918 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:33273
--local_ip_for_outbound_sockets=127.13.94.131
--tserver_master_addrs=127.13.94.190:42167
--webserver_port=41835
--webserver_interface=127.13.94.131
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:45253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.94.190:42167,127.13.94.189:34257
--tserver_master_addrs=127.13.94.190:42167,127.13.94.189:34257,127.13.94.188:43089 with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 30 08:01:17 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536077, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
Apr 30 08:01:17 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536077, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.189@KRBTEST.COM
Apr 30 08:01:17 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536077, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.188@KRBTEST.COM
I20260430 08:01:17.088840 21292 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.075146 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:54933 (local address 127.13.94.189:34257)
0430 08:01:17.075274 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.075278 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.075504 (+ 226us) server_negotiation.cc:408] Connection header received
0430 08:01:17.075660 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.075664 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.075719 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.075777 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.078829 (+ 3052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.079536 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.080481 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.080683 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.083502 (+ 2819us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.083523 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.083528 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.083563 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.086045 (+ 2482us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.086797 (+ 752us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.086803 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.086807 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.086870 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.087259 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.087263 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.087265 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.087470 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.087559 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.088528 (+ 969us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.088641 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 08:01:17.089031 21295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.073080 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:42475 (local address 127.13.94.188:43089)
0430 08:01:17.073247 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.073253 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.073857 (+ 604us) server_negotiation.cc:408] Connection header received
0430 08:01:17.075013 (+ 1156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.075018 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.075081 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.075139 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.078616 (+ 3477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.079339 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.080633 (+ 1294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.080846 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.083999 (+ 3153us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.084026 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.084028 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.084055 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.086045 (+ 1990us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.087421 (+ 1376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.087426 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.087429 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.087494 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.087818 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.087824 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.087827 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.088057 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.088142 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.088544 (+ 402us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.088688 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 08:01:17.089543 21567 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.074147 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.130:41967)
0430 08:01:17.074478 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.074495 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.074789 (+ 294us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.075551 (+ 762us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.075561 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.076271 (+ 710us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:17.078099 (+ 1828us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.078120 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.079262 (+ 1142us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.079269 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.079837 (+ 568us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.079848 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.080066 (+ 218us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.080820 (+ 754us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:17.080849 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:17.082940 (+ 2091us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:17.086828 (+ 3888us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.086831 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.086834 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.087044 (+ 210us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.087355 (+ 311us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.087358 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.087360 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.087404 (+ 44us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.087810 (+ 406us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:17.087813 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:17.087952 (+ 139us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.088263 (+ 311us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.088549 (+ 286us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"spinlock_wait_cycles":54528,"thread_start_us":120,"threads_started":1}
I20260430 08:01:17.089838 21566 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.073256 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.13.94.130:42475)
0430 08:01:17.073688 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.073727 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.074761 (+ 1034us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.075548 (+ 787us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.075559 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.076268 (+ 709us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:17.078100 (+ 1832us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.078120 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.079477 (+ 1357us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.079480 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.080180 (+ 700us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.080189 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.080971 (+ 782us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.081831 (+ 860us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:17.081844 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:17.083860 (+ 2016us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:17.087028 (+ 3168us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.087033 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.087037 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.087281 (+ 244us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.087628 (+ 347us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.087631 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.087636 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.087700 (+ 64us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.088226 (+ 526us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:17.088231 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:17.088335 (+ 104us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.089424 (+ 1089us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.089586 (+ 162us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"spinlock_wait_cycles":14848,"thread_start_us":116,"threads_started":1}
I20260430 08:01:17.090402 21568 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.074723 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.13.94.130:54933)
0430 08:01:17.075434 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.075447 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.075559 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.075917 (+ 358us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.075920 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.078399 (+ 2479us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:17.078694 (+ 295us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.078701 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.079652 (+ 951us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.079655 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.080361 (+ 706us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.080371 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.080845 (+ 474us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.081346 (+ 501us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:17.081366 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:17.083349 (+ 1983us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:17.086196 (+ 2847us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.086206 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.086224 (+ 18us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.086649 (+ 425us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.086995 (+ 346us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.086998 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.087000 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.087152 (+ 152us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.087616 (+ 464us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:17.087622 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:17.087952 (+ 330us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.088263 (+ 311us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.090236 (+ 1973us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":1719,"spinlock_wait_cycles":5120,"thread_start_us":81,"threads_started":1}
I20260430 08:01:17.090934 21294 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.072641 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:41967 (local address 127.13.94.190:42167)
0430 08:01:17.072846 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.072853 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.074574 (+ 1721us) server_negotiation.cc:408] Connection header received
0430 08:01:17.075017 (+ 443us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.075024 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.075093 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.075152 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.078300 (+ 3148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.079116 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.080373 (+ 1257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.080611 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.083157 (+ 2546us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.083184 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.083190 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.083230 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.086700 (+ 3470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.087177 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.087184 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.087189 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.087253 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.087503 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.087510 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.087515 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.087688 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.087772 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.090634 (+ 2862us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.090783 (+ 149us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260430 08:01:17.091444 21562 heartbeater.cc:344] Connected to a master server at 127.13.94.188:43089
I20260430 08:01:17.091576 21563 heartbeater.cc:344] Connected to a master server at 127.13.94.190:42167
I20260430 08:01:17.091722 21562 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:17.091742 21564 heartbeater.cc:344] Connected to a master server at 127.13.94.189:34257
I20260430 08:01:17.091809 21564 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:17.092412 21562 heartbeater.cc:507] Master 127.13.94.188:43089 requested a full tablet report, sending...
I20260430 08:01:17.092412 21564 heartbeater.cc:507] Master 127.13.94.189:34257 requested a full tablet report, sending...
I20260430 08:01:17.091722 21563 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:17.093231 21563 heartbeater.cc:507] Master 127.13.94.190:42167 requested a full tablet report, sending...
I20260430 08:01:17.093803 21046 ts_manager.cc:194] Registered new tserver with Master: 42439d79f4494fef9d7096a3e096a023 (127.13.94.130:34387)
I20260430 08:01:17.093909 21231 ts_manager.cc:194] Registered new tserver with Master: 42439d79f4494fef9d7096a3e096a023 (127.13.94.130:34387)
I20260430 08:01:17.094560 20433 ts_manager.cc:194] Re-registered known tserver with Master: 42439d79f4494fef9d7096a3e096a023 (127.13.94.130:34387)
I20260430 08:01:17.095391 20433 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:41967
W20260430 08:01:17.197535 21569 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:17.197831 21569 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:17.197932 21569 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:17.201583 21569 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:17.201690 21569 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:17.201823 21569 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:01:17.206905 21569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:45253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:33273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=41835
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:42167,127.13.94.189:34257,127.13.94.188:43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:17.208187 21569 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:17.209262 21569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:17.216408 21577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:17.216416 21575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:17.216406 21574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:17.216825 21569 server_base.cc:1061] running on GCE node
I20260430 08:01:17.217435 21569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:17.218075 21569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:17.219256 21569 hybrid_clock.cc:648] HybridClock initialized: now 1777536077219231 us; error 40 us; skew 500 ppm
Apr 30 08:01:17 dist-test-slave-9bxp krb5kdc[20387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536077, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:17.222826 21569 init.cc:377] Logged in from keytab as oryx/127.13.94.131@KRBTEST.COM (short username oryx)
I20260430 08:01:17.224129 21569 webserver.cc:492] Webserver started at http://127.13.94.131:41835/ using document root <none> and password file <none>
I20260430 08:01:17.224802 21569 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:17.224869 21569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:17.228500 21569 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:17.230572 21584 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:17.231700 21569 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:17.231858 21569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "7183343b3df947c6817050cb5ae11661"
format_stamp: "Formatted at 2026-04-30 08:01:14 on dist-test-slave-9bxp"
server_key: "30deb78fd66573e9ffa87321cd7b9fa8"
server_key_iv: "76e9f10027e476041d03df826ac1732e"
server_key_version: "encryptionkey@0"
I20260430 08:01:17.232336 21569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:17.254367 21569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:17.257853 21569 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:17.258107 21569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:17.258744 21569 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:17.259785 21569 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:17.259858 21569 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:17.259930 21569 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:17.259981 21569 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:17.270198 21569 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:33273
I20260430 08:01:17.270277 21697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:33273 every 8 connection(s)
I20260430 08:01:17.271309 21569 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:01:17.278458 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 21569
Apr 30 08:01:17 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536077, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
Apr 30 08:01:17 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536077, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.188@KRBTEST.COM
Apr 30 08:01:17 dist-test-slave-9bxp krb5kdc[20387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536077, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.189@KRBTEST.COM
I20260430 08:01:17.288276 21292 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.273140 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:40133 (local address 127.13.94.189:34257)
0430 08:01:17.273295 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.273299 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.274063 (+ 764us) server_negotiation.cc:408] Connection header received
0430 08:01:17.275563 (+ 1500us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.275567 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.275623 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.275709 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.277291 (+ 1582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.278040 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.278988 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.279188 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.283239 (+ 4051us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.283258 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.283262 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.283291 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.285496 (+ 2205us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.286321 (+ 825us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.286325 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.286327 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.286392 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.286779 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.286786 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.286790 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.286989 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.287095 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.287966 (+ 871us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.288088 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260430 08:01:17.289018 21703 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.273473 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.189:34257 (local address 127.13.94.131:40133)
0430 08:01:17.273915 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.273958 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.275018 (+ 1060us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.275773 (+ 755us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.275776 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.276445 (+ 669us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:17.277125 (+ 680us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.277131 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.278167 (+ 1036us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.278170 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.278843 (+ 673us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.278852 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.279353 (+ 501us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.280090 (+ 737us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:17.280104 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:17.283086 (+ 2982us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:17.285658 (+ 2572us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.285668 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.285687 (+ 19us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.286146 (+ 459us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.286497 (+ 351us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.286500 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.286503 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.286663 (+ 160us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.287145 (+ 482us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:17.287149 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:17.287476 (+ 327us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.287709 (+ 233us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.288517 (+ 808us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":609,"spinlock_wait_cycles":94080,"thread_start_us":143,"threads_started":1}
I20260430 08:01:17.289177 21294 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.273173 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:37757 (local address 127.13.94.190:42167)
0430 08:01:17.273310 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.273315 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.274063 (+ 748us) server_negotiation.cc:408] Connection header received
0430 08:01:17.275246 (+ 1183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.275250 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.275317 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.275422 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.277232 (+ 1810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.278049 (+ 817us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.278932 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.279143 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.283240 (+ 4097us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.283267 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.283270 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.283299 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.285789 (+ 2490us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.286350 (+ 561us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.286356 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.286358 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.286417 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.286752 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.286755 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.286758 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.286949 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.287043 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.288838 (+ 1795us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.289000 (+ 162us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260430 08:01:17.289386 21702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.273473 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.131:37757)
0430 08:01:17.273916 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.273959 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.275019 (+ 1060us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.275616 (+ 597us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.275629 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.276630 (+ 1001us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:17.277061 (+ 431us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.277074 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.278220 (+ 1146us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.278224 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.278775 (+ 551us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.278785 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.279296 (+ 511us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.279969 (+ 673us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:17.279990 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:17.283086 (+ 3096us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:17.285907 (+ 2821us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.285910 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.285913 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.286216 (+ 303us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.286528 (+ 312us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.286534 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.286539 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.286648 (+ 109us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.287099 (+ 451us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:17.287106 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:17.287470 (+ 364us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.287709 (+ 239us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.287988 (+ 279us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"spinlock_wait_cycles":21504,"thread_start_us":101,"threads_started":1}
I20260430 08:01:17.290426 21699 heartbeater.cc:344] Connected to a master server at 127.13.94.190:42167
I20260430 08:01:17.290477 21700 heartbeater.cc:344] Connected to a master server at 127.13.94.189:34257
I20260430 08:01:17.290690 21699 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:17.291327 21699 heartbeater.cc:507] Master 127.13.94.190:42167 requested a full tablet report, sending...
I20260430 08:01:17.290692 21700 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:17.292330 21700 heartbeater.cc:507] Master 127.13.94.189:34257 requested a full tablet report, sending...
I20260430 08:01:17.292568 20433 ts_manager.cc:194] Re-registered known tserver with Master: 7183343b3df947c6817050cb5ae11661 (127.13.94.131:33273)
I20260430 08:01:17.293329 21704 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.274432 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.188:43089 (local address 127.13.94.131:58157)
0430 08:01:17.274727 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.274744 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.275018 (+ 274us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.276074 (+ 1056us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.276077 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.276445 (+ 368us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:17.277114 (+ 669us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.277123 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.277915 (+ 792us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.277917 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.278442 (+ 525us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.278450 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.279044 (+ 594us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.279732 (+ 688us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:17.279751 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:17.282562 (+ 2811us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:17.288313 (+ 5751us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.288319 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.288325 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.288925 (+ 600us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.289798 (+ 873us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:17.289804 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:17.289806 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:17.289874 (+ 68us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:17.292840 (+ 2966us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:17.292844 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:17.292926 (+ 82us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.293028 (+ 102us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.293159 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"spinlock_wait_cycles":4864,"thread_start_us":100,"threads_started":1}
I20260430 08:01:17.293365 20433 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:37757
I20260430 08:01:17.293910 21295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.274564 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:58157 (local address 127.13.94.188:43089)
0430 08:01:17.274713 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.274716 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.274844 (+ 128us) server_negotiation.cc:408] Connection header received
0430 08:01:17.275824 (+ 980us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.275828 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.275887 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.275972 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.277293 (+ 1321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.277784 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.278583 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.278782 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.282783 (+ 4001us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.282804 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.282811 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.282846 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.288175 (+ 5329us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.289074 (+ 899us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.289078 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.289080 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.289138 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.290118 (+ 980us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.290121 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.290123 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.290296 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.290385 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.293635 (+ 3250us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.293760 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260430 08:01:17.293910 21046 ts_manager.cc:194] Registered new tserver with Master: 7183343b3df947c6817050cb5ae11661 (127.13.94.131:33273)
I20260430 08:01:17.295023 21698 heartbeater.cc:344] Connected to a master server at 127.13.94.188:43089
I20260430 08:01:17.295131 21698 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:17.295351 21698 heartbeater.cc:507] Master 127.13.94.188:43089 requested a full tablet report, sending...
I20260430 08:01:17.296185 21231 ts_manager.cc:194] Registered new tserver with Master: 7183343b3df947c6817050cb5ae11661 (127.13.94.131:33273)
I20260430 08:01:17.297052 21709 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.281321 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.13.94.190:42167)
0430 08:01:17.281636 (+ 315us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.281639 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.281888 (+ 249us) server_negotiation.cc:408] Connection header received
0430 08:01:17.283124 (+ 1236us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.283129 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.283200 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.283288 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.284685 (+ 1397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.285565 (+ 880us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.289686 (+ 4121us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.289942 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.290045 (+ 103us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.290062 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.290069 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.290094 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.293611 (+ 3517us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.294668 (+ 1057us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.294672 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.294674 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.294729 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.295222 (+ 493us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.295226 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.295228 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.295496 (+ 268us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.295646 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.296721 (+ 1075us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.296858 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260430 08:01:17.298235 21711 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.281447 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.13.94.189:34257)
0430 08:01:17.282188 (+ 741us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.282192 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.282395 (+ 203us) server_negotiation.cc:408] Connection header received
0430 08:01:17.282581 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.282585 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.282636 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.282700 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.285512 (+ 2812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.287222 (+ 1710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.288187 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.288383 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.291202 (+ 2819us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.291221 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.291224 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.291256 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.293095 (+ 1839us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.296238 (+ 3143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.296245 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.296246 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.296302 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.296578 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.296583 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.296588 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.296979 (+ 391us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.297116 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.297698 (+ 582us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.297823 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":79,"threads_started":1}
I20260430 08:01:17.300988 21713 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.282409 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.13.94.188:43089)
0430 08:01:17.291360 (+ 8951us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.291366 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.291380 (+ 14us) server_negotiation.cc:408] Connection header received
0430 08:01:17.291419 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.291423 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.291475 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.291575 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.292600 (+ 1025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.293659 (+ 1059us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.294755 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.294978 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.296638 (+ 1660us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.296658 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.296660 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.296686 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.299231 (+ 2545us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.299650 (+ 419us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.299656 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.299658 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.299710 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.300122 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.300124 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.300126 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.300274 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.300372 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.300627 (+ 255us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.300782 (+ 155us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8847,"thread_start_us":73,"threads_started":1}
I20260430 08:01:17.300263 20433 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:01:17.303180 20433 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:01:17.321117 21723 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.317042 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:33273 (local address 127.0.0.1:58528)
0430 08:01:17.317339 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.317356 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.317458 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.318198 (+ 740us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.318202 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.318225 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:17.318503 (+ 278us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.318511 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.319833 (+ 1322us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.319836 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.320666 (+ 830us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.320677 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.320831 (+ 154us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.320854 (+ 23us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.320907 (+ 53us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.320973 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":156,"threads_started":1}
I20260430 08:01:17.322165 21727 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.317731 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:36283 (local address 127.0.0.1:34810)
0430 08:01:17.318707 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.318721 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.318808 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.319345 (+ 537us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.319349 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.319369 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:17.319622 (+ 253us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.319630 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.320872 (+ 1242us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.320876 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.321811 (+ 935us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.321819 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.321924 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.321938 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.321988 (+ 50us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.322038 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"thread_start_us":92,"threads_started":1}
I20260430 08:01:17.322788 21725 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.318436 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:34387 (local address 127.0.0.1:37004)
0430 08:01:17.318899 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.318914 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.319005 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.319817 (+ 812us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.319822 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.319841 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:17.320089 (+ 248us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.320094 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.321509 (+ 1415us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.321512 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.322380 (+ 868us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.322391 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.322532 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.322549 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.322598 (+ 49us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.322647 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":97,"threads_started":1}
I20260430 08:01:17.322911 21726 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.317834 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.13.94.129:36283)
0430 08:01:17.318191 (+ 357us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.318197 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.318897 (+ 700us) server_negotiation.cc:408] Connection header received
0430 08:01:17.318990 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.318995 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.319190 (+ 195us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.319339 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:17.319746 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.320689 (+ 943us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.321954 (+ 1265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.322461 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.322577 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.322647 (+ 70us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.322731 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260430 08:01:17.323150 21724 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.317212 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.13.94.131:33273)
0430 08:01:17.317675 (+ 463us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.317682 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.317706 (+ 24us) server_negotiation.cc:408] Connection header received
0430 08:01:17.317802 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.317808 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.318028 (+ 220us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.318261 (+ 233us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:17.318670 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.319648 (+ 978us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.321714 (+ 2066us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.322445 (+ 731us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.322577 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.322647 (+ 70us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.322731 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":121,"threads_started":1}
I20260430 08:01:17.323505 21728 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.318837 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37004 (local address 127.13.94.130:34387)
0430 08:01:17.319236 (+ 399us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.319242 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.319257 (+ 15us) server_negotiation.cc:408] Connection header received
0430 08:01:17.319356 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.319361 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.319516 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.319649 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:17.320224 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.321376 (+ 1152us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.322547 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.323104 (+ 557us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.323207 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.323279 (+ 72us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.323359 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":133,"threads_started":1}
I20260430 08:01:17.325829 21360 tablet_service.cc:1511] Processing CreateTablet for tablet dd32937570a549119f3083ba7be57743 (DEFAULT_TABLE table=test-table [id=1bc429fb042b4ab088f9ee105b5c7046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:17.325871 21632 tablet_service.cc:1511] Processing CreateTablet for tablet dd32937570a549119f3083ba7be57743 (DEFAULT_TABLE table=test-table [id=1bc429fb042b4ab088f9ee105b5c7046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:17.326732 21360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd32937570a549119f3083ba7be57743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:17.326750 21632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd32937570a549119f3083ba7be57743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:17.326481 21496 tablet_service.cc:1511] Processing CreateTablet for tablet dd32937570a549119f3083ba7be57743 (DEFAULT_TABLE table=test-table [id=1bc429fb042b4ab088f9ee105b5c7046]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:17.327278 21496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd32937570a549119f3083ba7be57743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:17.333905 21731 tablet_bootstrap.cc:492] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: Bootstrap starting.
I20260430 08:01:17.334417 21729 tablet_bootstrap.cc:492] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: Bootstrap starting.
I20260430 08:01:17.334797 21730 tablet_bootstrap.cc:492] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: Bootstrap starting.
I20260430 08:01:17.336416 21731 tablet_bootstrap.cc:654] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:17.336477 21729 tablet_bootstrap.cc:654] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:17.336890 21730 tablet_bootstrap.cc:654] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:17.337384 21729 log.cc:826] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:17.337384 21731 log.cc:826] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:17.337633 21730 log.cc:826] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:17.339497 21730 tablet_bootstrap.cc:492] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: No bootstrap required, opened a new log
I20260430 08:01:17.339536 21729 tablet_bootstrap.cc:492] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: No bootstrap required, opened a new log
I20260430 08:01:17.339687 21730 ts_tablet_manager.cc:1403] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260430 08:01:17.339715 21729 ts_tablet_manager.cc:1403] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260430 08:01:17.341176 21731 tablet_bootstrap.cc:492] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: No bootstrap required, opened a new log
I20260430 08:01:17.341404 21731 ts_tablet_manager.cc:1403] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260430 08:01:17.342808 21729 raft_consensus.cc:359] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:17.342808 21730 raft_consensus.cc:359] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:17.343035 21729 raft_consensus.cc:385] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:17.343034 21730 raft_consensus.cc:385] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:17.343097 21730 raft_consensus.cc:740] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7183343b3df947c6817050cb5ae11661, State: Initialized, Role: FOLLOWER
I20260430 08:01:17.343106 21729 raft_consensus.cc:740] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3b88c84b80445f96a8debdb48b5fdb, State: Initialized, Role: FOLLOWER
I20260430 08:01:17.343575 21730 consensus_queue.cc:260] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:17.343801 21729 consensus_queue.cc:260] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:17.344455 21730 ts_tablet_manager.cc:1434] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 08:01:17.344642 21729 ts_tablet_manager.cc:1434] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 08:01:17.344766 21731 raft_consensus.cc:359] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:17.345000 21731 raft_consensus.cc:385] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:17.345079 21731 raft_consensus.cc:740] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42439d79f4494fef9d7096a3e096a023, State: Initialized, Role: FOLLOWER
I20260430 08:01:17.345505 21731 consensus_queue.cc:260] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:17.346014 21427 heartbeater.cc:499] Master 127.13.94.190:42167 was elected leader, sending a full tablet report...
I20260430 08:01:17.346662 21731 ts_tablet_manager.cc:1434] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260430 08:01:17.347793 21563 heartbeater.cc:499] Master 127.13.94.190:42167 was elected leader, sending a full tablet report...
I20260430 08:01:17.348284 21699 heartbeater.cc:499] Master 127.13.94.190:42167 was elected leader, sending a full tablet report...
W20260430 08:01:17.371183 21429 tablet.cc:2404] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:01:17.522619 21701 tablet.cc:2404] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:01:17.578997 21565 tablet.cc:2404] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:01:17.604328 21736 raft_consensus.cc:493] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:01:17.604569 21736 raft_consensus.cc:515] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:17.605914 21736 leader_election.cc:290] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7183343b3df947c6817050cb5ae11661 (127.13.94.131:33273), 42439d79f4494fef9d7096a3e096a023 (127.13.94.130:34387)
I20260430 08:01:17.609701 21738 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.606175 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:33273 (local address 127.13.94.129:35015)
0430 08:01:17.606556 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.606568 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.606692 (+ 124us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.606995 (+ 303us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.606998 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.607027 (+ 29us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:17.607352 (+ 325us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.607361 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.608382 (+ 1021us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.608386 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.609231 (+ 845us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.609240 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.609366 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.609383 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.609437 (+ 54us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.609491 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":125,"spinlock_wait_cycles":20608,"thread_start_us":89,"threads_started":1}
I20260430 08:01:17.609701 21739 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.606170 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:34387 (local address 127.13.94.129:36865)
0430 08:01:17.606539 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:17.606553 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:17.606692 (+ 139us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:17.606995 (+ 303us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:17.607003 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:17.607026 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:17.607352 (+ 326us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.607361 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.608432 (+ 1071us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.608436 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:17.609258 (+ 822us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:17.609266 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.609394 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.609411 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:01:17.609462 (+ 51us) client_negotiation.cc:241] Negotiation successful
0430 08:01:17.609521 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"spinlock_wait_cycles":12800,"thread_start_us":88,"threads_started":1}
I20260430 08:01:17.610133 21728 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.606294 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:36865 (local address 127.13.94.130:34387)
0430 08:01:17.606441 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.606446 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.606655 (+ 209us) server_negotiation.cc:408] Connection header received
0430 08:01:17.606809 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.606813 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.606871 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.606961 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:17.607480 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.608310 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.609382 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.609858 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.609900 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.609957 (+ 57us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.610022 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260430 08:01:17.610168 21724 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.606233 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:35015 (local address 127.13.94.131:33273)
0430 08:01:17.606387 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.606394 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.606651 (+ 257us) server_negotiation.cc:408] Connection header received
0430 08:01:17.606809 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.606813 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.606870 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.606961 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:17.607480 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.608261 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.609352 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.609847 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.609892 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.609948 (+ 56us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.610021 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260430 08:01:17.610747 21516 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd32937570a549119f3083ba7be57743" candidate_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42439d79f4494fef9d7096a3e096a023" is_pre_election: true
I20260430 08:01:17.610764 21652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd32937570a549119f3083ba7be57743" candidate_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7183343b3df947c6817050cb5ae11661" is_pre_election: true
I20260430 08:01:17.611021 21516 raft_consensus.cc:2468] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3b88c84b80445f96a8debdb48b5fdb in term 0.
I20260430 08:01:17.611032 21652 raft_consensus.cc:2468] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb3b88c84b80445f96a8debdb48b5fdb in term 0.
I20260430 08:01:17.611506 21315 leader_election.cc:304] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42439d79f4494fef9d7096a3e096a023, fb3b88c84b80445f96a8debdb48b5fdb; no voters:
I20260430 08:01:17.611799 21736 raft_consensus.cc:2804] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:01:17.611884 21736 raft_consensus.cc:493] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:01:17.611927 21736 raft_consensus.cc:3060] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:17.612900 21736 raft_consensus.cc:515] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:17.613270 21736 leader_election.cc:290] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [CANDIDATE]: Term 1 election: Requested vote from peers 7183343b3df947c6817050cb5ae11661 (127.13.94.131:33273), 42439d79f4494fef9d7096a3e096a023 (127.13.94.130:34387)
I20260430 08:01:17.613667 21516 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd32937570a549119f3083ba7be57743" candidate_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42439d79f4494fef9d7096a3e096a023"
I20260430 08:01:17.613675 21652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd32937570a549119f3083ba7be57743" candidate_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7183343b3df947c6817050cb5ae11661"
I20260430 08:01:17.613804 21516 raft_consensus.cc:3060] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:17.613821 21652 raft_consensus.cc:3060] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:17.614878 21516 raft_consensus.cc:2468] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3b88c84b80445f96a8debdb48b5fdb in term 1.
I20260430 08:01:17.615186 21652 raft_consensus.cc:2468] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb3b88c84b80445f96a8debdb48b5fdb in term 1.
I20260430 08:01:17.615250 21315 leader_election.cc:304] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42439d79f4494fef9d7096a3e096a023, fb3b88c84b80445f96a8debdb48b5fdb; no voters:
I20260430 08:01:17.615420 21736 raft_consensus.cc:2804] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:17.615607 21736 raft_consensus.cc:697] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 1 LEADER]: Becoming Leader. State: Replica: fb3b88c84b80445f96a8debdb48b5fdb, State: Running, Role: LEADER
I20260430 08:01:17.615948 21736 consensus_queue.cc:237] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:17.619555 20435 catalog_manager.cc:5671] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to fb3b88c84b80445f96a8debdb48b5fdb (127.13.94.129). New cstate: current_term: 1 leader_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } health_report { overall_health: UNKNOWN } } }
I20260430 08:01:17.689570 21726 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.685587 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.13.94.129:36283)
0430 08:01:17.685817 (+ 230us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.685821 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.685838 (+ 17us) server_negotiation.cc:408] Connection header received
0430 08:01:17.685966 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.685970 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.686083 (+ 113us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.686235 (+ 152us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:17.686787 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.687589 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.688396 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.688589 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.688768 (+ 179us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:17.689174 (+ 406us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:17.689314 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.689372 (+ 58us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.689422 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260430 08:01:17.695291 21516 raft_consensus.cc:1275] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [term 1 FOLLOWER]: Refusing update from remote peer fb3b88c84b80445f96a8debdb48b5fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:17.695315 21652 raft_consensus.cc:1275] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [term 1 FOLLOWER]: Refusing update from remote peer fb3b88c84b80445f96a8debdb48b5fdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:17.696022 21740 consensus_queue.cc:1048] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:17.696166 21736 consensus_queue.cc:1048] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:17.704595 21745 mvcc.cc:204] Tried to move back new op lower bound from 7280787774232350720 to 7280787773917999104. Current Snapshot: MvccSnapshot[applied={T|T < 7280787774232350720}]
I20260430 08:01:17.706977 21743 mvcc.cc:204] Tried to move back new op lower bound from 7280787774232350720 to 7280787773917999104. Current Snapshot: MvccSnapshot[applied={T|T < 7280787774232350720}]
I20260430 08:01:17.710592 21744 mvcc.cc:204] Tried to move back new op lower bound from 7280787774232350720 to 7280787773917999104. Current Snapshot: MvccSnapshot[applied={T|T < 7280787774232350720}]
I20260430 08:01:17.717655 20435 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:17.717804 20435 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:17.723140 20435 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 15f647c7facb44c89330f7899a1f3c99: Sending DeleteTablet for 3 replicas of tablet dd32937570a549119f3083ba7be57743
I20260430 08:01:17.724184 21632 tablet_service.cc:1558] Processing DeleteTablet for tablet dd32937570a549119f3083ba7be57743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:17 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:58528
I20260430 08:01:17.724221 21360 tablet_service.cc:1558] Processing DeleteTablet for tablet dd32937570a549119f3083ba7be57743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:17 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:34810
I20260430 08:01:17.724820 21753 tablet_replica.cc:333] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: stopping tablet replica
I20260430 08:01:17.725142 21754 tablet_replica.cc:333] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: stopping tablet replica
I20260430 08:01:17.725234 21496 tablet_service.cc:1558] Processing DeleteTablet for tablet dd32937570a549119f3083ba7be57743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:17 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:37004
I20260430 08:01:17.725298 21753 raft_consensus.cc:2243] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:01:17.725376 21754 raft_consensus.cc:2243] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:17.725721 21754 raft_consensus.cc:2272] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:17.725914 21753 raft_consensus.cc:2272] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:17.727304 21754 ts_tablet_manager.cc:1916] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:17.727955 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 20403
I20260430 08:01:17.728456 21753 ts_tablet_manager.cc:1916] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:17.730367 21754 ts_tablet_manager.cc:1929] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:17.730468 21754 log.cc:1199] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/dd32937570a549119f3083ba7be57743
I20260430 08:01:17.730795 21754 ts_tablet_manager.cc:1950] T dd32937570a549119f3083ba7be57743 P 7183343b3df947c6817050cb5ae11661: Deleting consensus metadata
I20260430 08:01:17.731302 21753 ts_tablet_manager.cc:1929] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:17.731448 21753 log.cc:1199] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/dd32937570a549119f3083ba7be57743
I20260430 08:01:17.731814 21753 ts_tablet_manager.cc:1950] T dd32937570a549119f3083ba7be57743 P fb3b88c84b80445f96a8debdb48b5fdb: Deleting consensus metadata
I20260430 08:01:17.735404 21755 tablet_replica.cc:333] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: stopping tablet replica
I20260430 08:01:17.735759 21755 raft_consensus.cc:2243] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:17.736105 21755 raft_consensus.cc:2272] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:17.737574 21755 ts_tablet_manager.cc:1916] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260430 08:01:17.738514 21316 connection.cc:570] server connection from 127.0.0.1:34810 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260430 08:01:17.739007 21585 connection.cc:570] server connection from 127.0.0.1:58528 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260430 08:01:17.740172 21755 ts_tablet_manager.cc:1929] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:17.740278 21755 log.cc:1199] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/dd32937570a549119f3083ba7be57743
I20260430 08:01:17.740561 21755 ts_tablet_manager.cc:1950] T dd32937570a549119f3083ba7be57743 P 42439d79f4494fef9d7096a3e096a023: Deleting consensus metadata
W20260430 08:01:17.741730 21450 connection.cc:463] server connection from 127.0.0.1:37004 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37004 (request call id 1) could send its response
I20260430 08:01:17.751438 21711 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.742384 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.13.94.189:34257)
0430 08:01:17.742540 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.742546 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.742718 (+ 172us) server_negotiation.cc:408] Connection header received
0430 08:01:17.742954 (+ 236us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.742959 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.743020 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.743077 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.744310 (+ 1233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.745241 (+ 931us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.746113 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.746347 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.747609 (+ 1262us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.747636 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.747639 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.747665 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.749560 (+ 1895us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.750135 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.750139 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.750141 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.750197 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.750488 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.750494 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.750498 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.750670 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.750808 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.751117 (+ 309us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.751258 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260430 08:01:17.752326 21713 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:17.742785 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.13.94.188:43089)
0430 08:01:17.742934 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:17.742939 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:17.743294 (+ 355us) server_negotiation.cc:408] Connection header received
0430 08:01:17.743465 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:17.743471 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:17.743533 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:17.743592 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:17.745019 (+ 1427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.745689 (+ 670us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:17.747033 (+ 1344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:17.747280 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:17.748594 (+ 1314us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:17.748619 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:17.748622 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:17.748657 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:17.750430 (+ 1773us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.750919 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.750923 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.750925 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.750979 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:17.751241 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:17.751244 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:17.751246 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:17.751422 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:17.751933 (+ 511us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:17.752019 (+ 86us) server_negotiation.cc:300] Negotiation successful
0430 08:01:17.752147 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260430 08:01:18.708562 21763 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:18.708085 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.131:51323)
0430 08:01:18.708398 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:18.708469 (+ 71us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":146,"threads_started":1}
W20260430 08:01:18.708900 21699 heartbeater.cc:646] Failed to heartbeat to 127.13.94.190:42167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
I20260430 08:01:18.710891 21764 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:18.710403 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.129:40251)
0430 08:01:18.710718 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:18.710798 (+ 80us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
W20260430 08:01:18.711360 21427 heartbeater.cc:646] Failed to heartbeat to 127.13.94.190:42167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
I20260430 08:01:18.713332 21765 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:18.712961 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.130:55045)
0430 08:01:18.713183 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:18.713246 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
W20260430 08:01:18.713632 21563 heartbeater.cc:646] Failed to heartbeat to 127.13.94.190:42167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
I20260430 08:01:19.223250 21766 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15f647c7facb44c89330f7899a1f3c99)
I20260430 08:01:19.223512 21766 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } }
I20260430 08:01:19.223664 21767 raft_consensus.cc:493] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 15f647c7facb44c89330f7899a1f3c99)
I20260430 08:01:19.223862 21767 raft_consensus.cc:515] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } }
I20260430 08:01:19.224655 21766 leader_election.cc:290] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167), e8c26e5434bf49f7afca8e447329f514 (127.13.94.189:34257)
I20260430 08:01:19.224841 21767 leader_election.cc:290] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167), 4887df80ff3040bb865cde1d4ed64d93 (127.13.94.188:43089)
I20260430 08:01:19.225276 21056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4887df80ff3040bb865cde1d4ed64d93" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8c26e5434bf49f7afca8e447329f514" is_pre_election: true
I20260430 08:01:19.225277 21241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8c26e5434bf49f7afca8e447329f514" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4887df80ff3040bb865cde1d4ed64d93" is_pre_election: true
I20260430 08:01:19.225292 21768 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.224805 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42988)
0430 08:01:19.225102 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.225172 (+ 70us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":64,"threads_started":1}
I20260430 08:01:19.225431 21769 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.225006 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:42990)
0430 08:01:19.225277 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.225337 (+ 60us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":53,"threads_started":1}
I20260430 08:01:19.225461 21056 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4887df80ff3040bb865cde1d4ed64d93 in term 1.
I20260430 08:01:19.225461 21241 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8c26e5434bf49f7afca8e447329f514 in term 1.
W20260430 08:01:19.225538 21033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
W20260430 08:01:19.225466 21218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
I20260430 08:01:19.226104 21217 leader_election.cc:304] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4887df80ff3040bb865cde1d4ed64d93, e8c26e5434bf49f7afca8e447329f514; no voters:
I20260430 08:01:19.226095 21032 leader_election.cc:304] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4887df80ff3040bb865cde1d4ed64d93, e8c26e5434bf49f7afca8e447329f514; no voters:
I20260430 08:01:19.226301 21766 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 08:01:19.226365 21766 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 15f647c7facb44c89330f7899a1f3c99)
I20260430 08:01:19.226406 21766 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 1 FOLLOWER]: Advancing to term 2
I20260430 08:01:19.226424 21767 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 08:01:19.226480 21767 raft_consensus.cc:493] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 15f647c7facb44c89330f7899a1f3c99)
I20260430 08:01:19.226521 21767 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 1 FOLLOWER]: Advancing to term 2
I20260430 08:01:19.226583 21768 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.226304 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43004)
0430 08:01:19.226462 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.226509 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260430 08:01:19.226729 21769 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.226442 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43008)
0430 08:01:19.226621 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.226663 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260430 08:01:19.226699 21218 leader_election.cc:336] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167): Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
W20260430 08:01:19.226805 21033 leader_election.cc:336] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167): Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
I20260430 08:01:19.227555 21766 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } }
I20260430 08:01:19.227607 21767 raft_consensus.cc:515] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } }
I20260430 08:01:19.227931 21766 leader_election.cc:290] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 2 election: Requested vote from peers 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167), e8c26e5434bf49f7afca8e447329f514 (127.13.94.189:34257)
I20260430 08:01:19.228230 21767 leader_election.cc:290] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [CANDIDATE]: Term 2 election: Requested vote from peers 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167), 4887df80ff3040bb865cde1d4ed64d93 (127.13.94.188:43089)
I20260430 08:01:19.228296 21768 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.228034 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43022)
0430 08:01:19.228180 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.228226 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260430 08:01:19.228279 21056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4887df80ff3040bb865cde1d4ed64d93" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8c26e5434bf49f7afca8e447329f514"
I20260430 08:01:19.228433 21056 raft_consensus.cc:2393] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4887df80ff3040bb865cde1d4ed64d93 in current term 2: Already voted for candidate e8c26e5434bf49f7afca8e447329f514 in this term.
I20260430 08:01:19.228531 21241 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8c26e5434bf49f7afca8e447329f514" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4887df80ff3040bb865cde1d4ed64d93"
I20260430 08:01:19.228682 21241 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e8c26e5434bf49f7afca8e447329f514 in current term 2: Already voted for candidate 4887df80ff3040bb865cde1d4ed64d93 in this term.
I20260430 08:01:19.228881 21769 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.228034 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43038)
0430 08:01:19.228537 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.228816 (+ 279us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260430 08:01:19.228983 21768 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.228795 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43040)
0430 08:01:19.228883 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.228928 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260430 08:01:19.229070 21218 leader_election.cc:336] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167): Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
I20260430 08:01:19.229143 21218 leader_election.cc:304] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4887df80ff3040bb865cde1d4ed64d93; no voters: 15f647c7facb44c89330f7899a1f3c99, e8c26e5434bf49f7afca8e447329f514
I20260430 08:01:19.229297 21766 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 08:01:19.229585 21769 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.229359 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43048)
0430 08:01:19.229500 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.229536 (+ 36us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260430 08:01:19.229691 21033 leader_election.cc:336] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167): Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
I20260430 08:01:19.229755 21033 leader_election.cc:304] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e8c26e5434bf49f7afca8e447329f514; no voters: 15f647c7facb44c89330f7899a1f3c99, 4887df80ff3040bb865cde1d4ed64d93
I20260430 08:01:19.229883 21767 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 08:01:19.710045 21772 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.709494 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.131:33549)
0430 08:01:19.709826 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.709900 (+ 74us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
I20260430 08:01:19.712491 21773 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.712045 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.129:46827)
0430 08:01:19.712341 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.712399 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":124,"threads_started":1}
I20260430 08:01:19.714573 21774 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:19.714136 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.130:57765)
0430 08:01:19.714383 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:19.714432 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260430 08:01:20.711342 21775 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.710852 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.131:42871)
0430 08:01:20.711148 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.711213 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260430 08:01:20.713512 21776 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.713098 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.129:48485)
0430 08:01:20.713328 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.713389 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260430 08:01:20.715624 21777 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.715226 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.130:45813)
0430 08:01:20.715451 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.715524 (+ 73us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260430 08:01:20.787875 21778 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:01:20.788019 21778 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } }
I20260430 08:01:20.788626 21778 leader_election.cc:290] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167), e8c26e5434bf49f7afca8e447329f514 (127.13.94.189:34257)
I20260430 08:01:20.789223 21056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4887df80ff3040bb865cde1d4ed64d93" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8c26e5434bf49f7afca8e447329f514" is_pre_election: true
I20260430 08:01:20.789305 21779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.788858 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43244)
0430 08:01:20.789130 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.789197 (+ 67us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260430 08:01:20.789387 21056 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4887df80ff3040bb865cde1d4ed64d93 in term 2.
I20260430 08:01:20.789830 21217 leader_election.cc:304] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4887df80ff3040bb865cde1d4ed64d93, e8c26e5434bf49f7afca8e447329f514; no voters:
I20260430 08:01:20.790038 21778 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 08:01:20.790110 21778 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:01:20.790153 21778 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 2 FOLLOWER]: Advancing to term 3
I20260430 08:01:20.790270 21779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.790012 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43260)
0430 08:01:20.790126 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.790195 (+ 69us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260430 08:01:20.790382 21218 leader_election.cc:336] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167): Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
I20260430 08:01:20.791215 21778 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } }
I20260430 08:01:20.791592 21778 leader_election.cc:290] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 3 election: Requested vote from peers 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167), e8c26e5434bf49f7afca8e447329f514 (127.13.94.189:34257)
I20260430 08:01:20.791970 21779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.791706 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43274)
0430 08:01:20.791849 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.791911 (+ 62us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260430 08:01:20.791967 21056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4887df80ff3040bb865cde1d4ed64d93" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e8c26e5434bf49f7afca8e447329f514"
I20260430 08:01:20.792083 21056 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 2 FOLLOWER]: Advancing to term 3
I20260430 08:01:20.792665 21779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.792451 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43278)
0430 08:01:20.792561 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.792603 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260430 08:01:20.792793 21218 leader_election.cc:336] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167): Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
I20260430 08:01:20.793234 21056 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4887df80ff3040bb865cde1d4ed64d93 in term 3.
I20260430 08:01:20.793550 21217 leader_election.cc:304] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4887df80ff3040bb865cde1d4ed64d93, e8c26e5434bf49f7afca8e447329f514; no voters: 15f647c7facb44c89330f7899a1f3c99
I20260430 08:01:20.793723 21778 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 08:01:20.793915 21778 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [term 3 LEADER]: Becoming Leader. State: Replica: 4887df80ff3040bb865cde1d4ed64d93, State: Running, Role: LEADER
I20260430 08:01:20.794152 21778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } }
I20260430 08:01:20.795395 21781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4887df80ff3040bb865cde1d4ed64d93. Latest consensus state: current_term: 3 leader_uuid: "4887df80ff3040bb865cde1d4ed64d93" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:20.795485 21781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:20.795765 21783 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:20.797817 21783 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:20.798158 21783 catalog_manager.cc:1269] Loaded cluster ID: 85a9016ecce6489d9d2fa6d395880450
I20260430 08:01:20.798200 21783 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:20.798669 21783 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:20.799012 21783 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Loaded TSK: 0
I20260430 08:01:20.799530 21783 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 08:01:20.815816 21231 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:01:20.818300 21231 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:01:20.821612 21056 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [term 3 FOLLOWER]: Refusing update from remote peer 4887df80ff3040bb865cde1d4ed64d93: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260430 08:01:20.821676 21779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.821224 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43302)
0430 08:01:20.821561 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.821605 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269}
I20260430 08:01:20.822166 21778 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260430 08:01:20.822409 21779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.822193 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.0.0.1:43310)
0430 08:01:20.822306 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.822353 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260430 08:01:20.822539 21218 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 -> Peer 15f647c7facb44c89330f7899a1f3c99 (127.13.94.190:42167): Couldn't send request to peer 15f647c7facb44c89330f7899a1f3c99. Status: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260430 08:01:20.823745 21785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4887df80ff3040bb865cde1d4ed64d93. Latest consensus state: current_term: 3 leader_uuid: "4887df80ff3040bb865cde1d4ed64d93" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:20.823858 21785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 08:01:20.825251 21781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4887df80ff3040bb865cde1d4ed64d93" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:20.825372 21781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:20.825405 21785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4887df80ff3040bb865cde1d4ed64d93" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "15f647c7facb44c89330f7899a1f3c99" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 42167 } } peers { permanent_uuid: "e8c26e5434bf49f7afca8e447329f514" member_type: VOTER last_known_addr { host: "127.13.94.189" port: 34257 } attrs { promote: false } } peers { permanent_uuid: "4887df80ff3040bb865cde1d4ed64d93" member_type: VOTER last_known_addr { host: "127.13.94.188" port: 43089 } attrs { promote: false } } }
I20260430 08:01:20.825512 21785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c26e5434bf49f7afca8e447329f514 [sys.catalog]: This master's current role is: FOLLOWER
I20260430 08:01:20.835708 21779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.831901 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:33273 (local address 127.0.0.1:58530)
0430 08:01:20.832056 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.832071 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:20.832170 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:20.833307 (+ 1137us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:20.833312 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:20.833335 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:20.833599 (+ 264us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:20.833607 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.834391 (+ 784us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:20.834397 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:20.835300 (+ 903us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:20.835309 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.835424 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:20.835442 (+ 18us) client_negotiation.cc:770] Sending connection context
0430 08:01:20.835497 (+ 55us) client_negotiation.cc:241] Negotiation successful
0430 08:01:20.835568 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260430 08:01:20.836468 21790 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.832045 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.13.94.131:33273)
0430 08:01:20.832519 (+ 474us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:20.832524 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:20.832540 (+ 16us) server_negotiation.cc:408] Connection header received
0430 08:01:20.832601 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:20.832608 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:20.832667 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:20.832812 (+ 145us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:20.833745 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.834277 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:20.835442 (+ 1165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.836135 (+ 693us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:20.836173 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:20.836226 (+ 53us) server_negotiation.cc:300] Negotiation successful
0430 08:01:20.836306 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":107,"threads_started":1}
I20260430 08:01:20.836735 21791 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.832282 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:36283 (local address 127.0.0.1:34822)
0430 08:01:20.833021 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.833041 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:20.833116 (+ 75us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:20.833394 (+ 278us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:20.833397 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:20.833419 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:20.833684 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:20.833694 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.834843 (+ 1149us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:20.834847 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:20.835699 (+ 852us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:20.835708 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.836396 (+ 688us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:20.836408 (+ 12us) client_negotiation.cc:770] Sending connection context
0430 08:01:20.836490 (+ 82us) client_negotiation.cc:241] Negotiation successful
0430 08:01:20.836552 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":123,"threads_started":1}
I20260430 08:01:20.836791 21793 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.832690 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.13.94.129:36283)
0430 08:01:20.832976 (+ 286us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:20.832981 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:20.833111 (+ 130us) server_negotiation.cc:408] Connection header received
0430 08:01:20.833192 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:20.833196 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:20.833253 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:20.833351 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:20.833805 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.834550 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:20.835849 (+ 1299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.836307 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:20.836337 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:20.836543 (+ 206us) server_negotiation.cc:300] Negotiation successful
0430 08:01:20.836618 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260430 08:01:20.837260 21632 tablet_service.cc:1511] Processing CreateTablet for tablet a2623323d0f840428d0194bc45b42a8a (DEFAULT_TABLE table=test-table [id=3d30e08695ab4e939f720ae5da14abc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:20.837530 21794 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.833394 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.13.94.130:34387)
0430 08:01:20.833707 (+ 313us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:20.833713 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:20.834072 (+ 359us) server_negotiation.cc:408] Connection header received
0430 08:01:20.834122 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:20.834125 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:20.834174 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:20.834245 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:20.834691 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.835427 (+ 736us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:20.836791 (+ 1364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.837254 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:20.837290 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:20.837339 (+ 49us) server_negotiation.cc:300] Negotiation successful
0430 08:01:20.837391 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260430 08:01:20.837570 21632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2623323d0f840428d0194bc45b42a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:20.837678 21360 tablet_service.cc:1511] Processing CreateTablet for tablet a2623323d0f840428d0194bc45b42a8a (DEFAULT_TABLE table=test-table [id=3d30e08695ab4e939f720ae5da14abc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:20.837764 21792 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.832636 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:34387 (local address 127.0.0.1:37006)
0430 08:01:20.833911 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.833923 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:20.834001 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:20.834333 (+ 332us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:20.834336 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:20.834352 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:20.834569 (+ 217us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:20.834576 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.835551 (+ 975us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:20.835554 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:20.836387 (+ 833us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:20.836395 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:20.836555 (+ 160us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:20.836568 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 08:01:20.836616 (+ 48us) client_negotiation.cc:241] Negotiation successful
0430 08:01:20.836663 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"thread_start_us":112,"threads_started":1}
I20260430 08:01:20.837955 21360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2623323d0f840428d0194bc45b42a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:20.838303 21496 tablet_service.cc:1511] Processing CreateTablet for tablet a2623323d0f840428d0194bc45b42a8a (DEFAULT_TABLE table=test-table [id=3d30e08695ab4e939f720ae5da14abc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:20.838544 21496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2623323d0f840428d0194bc45b42a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:20.839752 21795 tablet_bootstrap.cc:492] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb: Bootstrap starting.
I20260430 08:01:20.840277 21796 tablet_bootstrap.cc:492] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: Bootstrap starting.
I20260430 08:01:20.841410 21796 tablet_bootstrap.cc:654] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:20.841478 21795 tablet_bootstrap.cc:654] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:20.842128 21797 tablet_bootstrap.cc:492] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: Bootstrap starting.
I20260430 08:01:20.842733 21795 tablet_bootstrap.cc:492] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb: No bootstrap required, opened a new log
I20260430 08:01:20.842839 21795 ts_tablet_manager.cc:1403] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260430 08:01:20.843039 21797 tablet_bootstrap.cc:654] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:20.843039 21796 tablet_bootstrap.cc:492] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: No bootstrap required, opened a new log
I20260430 08:01:20.843130 21796 ts_tablet_manager.cc:1403] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 08:01:20.843462 21795 raft_consensus.cc:359] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:20.843633 21795 raft_consensus.cc:385] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:20.843688 21795 raft_consensus.cc:740] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3b88c84b80445f96a8debdb48b5fdb, State: Initialized, Role: FOLLOWER
I20260430 08:01:20.843667 21796 raft_consensus.cc:359] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:20.843776 21796 raft_consensus.cc:385] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:20.843806 21796 raft_consensus.cc:740] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7183343b3df947c6817050cb5ae11661, State: Initialized, Role: FOLLOWER
I20260430 08:01:20.843860 21795 consensus_queue.cc:260] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:20.843997 21796 consensus_queue.cc:260] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:20.844211 21795 ts_tablet_manager.cc:1434] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260430 08:01:20.844439 21797 tablet_bootstrap.cc:492] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: No bootstrap required, opened a new log
I20260430 08:01:20.844551 21797 ts_tablet_manager.cc:1403] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:20.845412 21776 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.845097 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.129:55097)
0430 08:01:20.845262 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.845329 (+ 67us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260430 08:01:20.845623 21797 raft_consensus.cc:359] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:20.845743 21797 raft_consensus.cc:385] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:20.845811 21797 raft_consensus.cc:740] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42439d79f4494fef9d7096a3e096a023, State: Initialized, Role: FOLLOWER
I20260430 08:01:20.845841 21796 ts_tablet_manager.cc:1434] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260430 08:01:20.846019 21797 consensus_queue.cc:260] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:20.846645 21775 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.846351 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.131:38505)
0430 08:01:20.846527 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.846574 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260430 08:01:20.847527 21776 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.847255 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.129:50383)
0430 08:01:20.847397 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.847451 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260430 08:01:20.848186 21797 ts_tablet_manager.cc:1434] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260430 08:01:20.848263 21426 heartbeater.cc:499] Master 127.13.94.188:43089 was elected leader, sending a full tablet report...
I20260430 08:01:20.848990 21777 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.848723 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.130:44133)
0430 08:01:20.848879 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.848922 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260430 08:01:20.849558 21698 heartbeater.cc:499] Master 127.13.94.188:43089 was elected leader, sending a full tablet report...
I20260430 08:01:20.849792 21777 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.849561 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.130:47827)
0430 08:01:20.849683 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.849725 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260430 08:01:20.850503 21775 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:20.850155 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.131:38097)
0430 08:01:20.850288 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:20.850337 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260430 08:01:20.850587 21562 heartbeater.cc:499] Master 127.13.94.188:43089 was elected leader, sending a full tablet report...
I20260430 08:01:21.009924 21802 raft_consensus.cc:493] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:01:21.010080 21802 raft_consensus.cc:515] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:21.011258 21802 leader_election.cc:290] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb3b88c84b80445f96a8debdb48b5fdb (127.13.94.129:36283), 42439d79f4494fef9d7096a3e096a023 (127.13.94.130:34387)
I20260430 08:01:21.014740 21775 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.011585 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:34387 (local address 127.13.94.131:43765)
0430 08:01:21.011710 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:21.011727 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:21.011841 (+ 114us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:21.012179 (+ 338us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:21.012183 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:21.012204 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:21.012509 (+ 305us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:21.012520 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.013554 (+ 1034us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:21.013559 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:21.014252 (+ 693us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:21.014260 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.014415 (+ 155us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:21.014453 (+ 38us) client_negotiation.cc:770] Sending connection context
0430 08:01:21.014516 (+ 63us) client_negotiation.cc:241] Negotiation successful
0430 08:01:21.014602 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260430 08:01:21.015024 21804 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.011585 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:36283 (local address 127.13.94.131:45121)
0430 08:01:21.012243 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:21.012256 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:21.012339 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:21.012613 (+ 274us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:21.012617 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:21.012644 (+ 27us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:21.012960 (+ 316us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:21.012967 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.013902 (+ 935us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:21.013906 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:21.014615 (+ 709us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:21.014623 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.014776 (+ 153us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:21.014790 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 08:01:21.014841 (+ 51us) client_negotiation.cc:241] Negotiation successful
0430 08:01:21.014889 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":327,"thread_start_us":137,"threads_started":1}
I20260430 08:01:21.015179 21794 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.011740 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:43765 (local address 127.13.94.130:34387)
0430 08:01:21.011886 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:21.011910 (+ 24us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:21.011929 (+ 19us) server_negotiation.cc:408] Connection header received
0430 08:01:21.011988 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:21.011991 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:21.012049 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:21.012163 (+ 114us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:21.012655 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.013392 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:21.014397 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.014885 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:21.014929 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:21.015004 (+ 75us) server_negotiation.cc:300] Negotiation successful
0430 08:01:21.015074 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 08:01:21.015569 21793 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.011857 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:45121 (local address 127.13.94.129:36283)
0430 08:01:21.012009 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:21.012013 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:21.012338 (+ 325us) server_negotiation.cc:408] Connection header received
0430 08:01:21.012420 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:21.012423 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:21.012473 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:21.012560 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:21.013094 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.013756 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:21.014763 (+ 1007us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.015264 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:21.015302 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:21.015357 (+ 55us) server_negotiation.cc:300] Negotiation successful
0430 08:01:21.015434 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260430 08:01:21.015656 21516 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2623323d0f840428d0194bc45b42a8a" candidate_uuid: "7183343b3df947c6817050cb5ae11661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42439d79f4494fef9d7096a3e096a023" is_pre_election: true
I20260430 08:01:21.015836 21516 raft_consensus.cc:2468] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7183343b3df947c6817050cb5ae11661 in term 0.
I20260430 08:01:21.016148 21380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2623323d0f840428d0194bc45b42a8a" candidate_uuid: "7183343b3df947c6817050cb5ae11661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" is_pre_election: true
I20260430 08:01:21.016360 21380 raft_consensus.cc:2468] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7183343b3df947c6817050cb5ae11661 in term 0.
I20260430 08:01:21.016408 21587 leader_election.cc:304] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42439d79f4494fef9d7096a3e096a023, 7183343b3df947c6817050cb5ae11661; no voters:
I20260430 08:01:21.016680 21802 raft_consensus.cc:2804] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:01:21.016775 21802 raft_consensus.cc:493] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:01:21.016815 21802 raft_consensus.cc:3060] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:21.017722 21802 raft_consensus.cc:515] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:21.018263 21802 leader_election.cc:290] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [CANDIDATE]: Term 1 election: Requested vote from peers fb3b88c84b80445f96a8debdb48b5fdb (127.13.94.129:36283), 42439d79f4494fef9d7096a3e096a023 (127.13.94.130:34387)
I20260430 08:01:21.018743 21380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2623323d0f840428d0194bc45b42a8a" candidate_uuid: "7183343b3df947c6817050cb5ae11661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3b88c84b80445f96a8debdb48b5fdb"
I20260430 08:01:21.018850 21516 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2623323d0f840428d0194bc45b42a8a" candidate_uuid: "7183343b3df947c6817050cb5ae11661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42439d79f4494fef9d7096a3e096a023"
I20260430 08:01:21.018903 21380 raft_consensus.cc:3060] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:21.019019 21516 raft_consensus.cc:3060] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:21.019886 21516 raft_consensus.cc:2468] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7183343b3df947c6817050cb5ae11661 in term 1.
I20260430 08:01:21.019888 21380 raft_consensus.cc:2468] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7183343b3df947c6817050cb5ae11661 in term 1.
I20260430 08:01:21.020210 21587 leader_election.cc:304] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42439d79f4494fef9d7096a3e096a023, 7183343b3df947c6817050cb5ae11661; no voters:
I20260430 08:01:21.020404 21802 raft_consensus.cc:2804] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:21.020676 21802 raft_consensus.cc:697] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 1 LEADER]: Becoming Leader. State: Replica: 7183343b3df947c6817050cb5ae11661, State: Running, Role: LEADER
I20260430 08:01:21.020967 21802 consensus_queue.cc:237] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } }
I20260430 08:01:21.021423 21804 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.021180 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.131:50781)
0430 08:01:21.021309 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:21.021357 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260430 08:01:21.024595 21231 catalog_manager.cc:5671] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7183343b3df947c6817050cb5ae11661 (127.13.94.131). New cstate: current_term: 1 leader_uuid: "7183343b3df947c6817050cb5ae11661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7183343b3df947c6817050cb5ae11661" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 33273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 } health_report { overall_health: UNKNOWN } } }
I20260430 08:01:21.078938 21790 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.075739 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.13.94.131:33273)
0430 08:01:21.075913 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:21.075918 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:21.075932 (+ 14us) server_negotiation.cc:408] Connection header received
0430 08:01:21.076105 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:21.076108 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:21.076163 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:21.076274 (+ 111us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:21.076627 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.077199 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:21.077865 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:21.078053 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:21.078102 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:21.078494 (+ 392us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:21.078618 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:21.078737 (+ 119us) server_negotiation.cc:300] Negotiation successful
0430 08:01:21.078784 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260430 08:01:21.081967 21516 raft_consensus.cc:1275] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [term 1 FOLLOWER]: Refusing update from remote peer 7183343b3df947c6817050cb5ae11661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:21.082325 21380 raft_consensus.cc:1275] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb [term 1 FOLLOWER]: Refusing update from remote peer 7183343b3df947c6817050cb5ae11661: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:21.082582 21805 consensus_queue.cc:1048] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42439d79f4494fef9d7096a3e096a023" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 34387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:21.083016 21802 consensus_queue.cc:1048] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3b88c84b80445f96a8debdb48b5fdb" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:21.085615 21777 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.085291 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.130:54111)
0430 08:01:21.085474 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:21.085528 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260430 08:01:21.086386 21776 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.086120 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.129:45677)
0430 08:01:21.086268 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:21.086315 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260430 08:01:21.089627 21804 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.089205 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.131:36043)
0430 08:01:21.089499 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:21.089549 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260430 08:01:21.090636 21777 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.089236 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.130:37669)
0430 08:01:21.090501 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:21.090549 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1187}
I20260430 08:01:21.092540 21776 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.092266 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:42167 (local address 127.13.94.129:34417)
0430 08:01:21.092410 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:21.092453 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260430 08:01:21.096025 21231 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:21.096192 21231 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:21.100539 21231 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4887df80ff3040bb865cde1d4ed64d93: Sending DeleteTablet for 3 replicas of tablet a2623323d0f840428d0194bc45b42a8a
I20260430 08:01:21.101361 21496 tablet_service.cc:1558] Processing DeleteTablet for tablet a2623323d0f840428d0194bc45b42a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:21 UTC) from {username='oryx', principal='oryx/127.13.94.188@KRBTEST.COM'} at 127.0.0.1:37006
I20260430 08:01:21.101372 21632 tablet_service.cc:1558] Processing DeleteTablet for tablet a2623323d0f840428d0194bc45b42a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:21 UTC) from {username='oryx', principal='oryx/127.13.94.188@KRBTEST.COM'} at 127.0.0.1:58530
I20260430 08:01:21.101419 21360 tablet_service.cc:1558] Processing DeleteTablet for tablet a2623323d0f840428d0194bc45b42a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:21 UTC) from {username='oryx', principal='oryx/127.13.94.188@KRBTEST.COM'} at 127.0.0.1:34822
I20260430 08:01:21.101778 21818 tablet_replica.cc:333] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: stopping tablet replica
I20260430 08:01:21.101907 21818 raft_consensus.cc:2243] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:01:21.102216 21818 raft_consensus.cc:2272] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:21.102531 21820 tablet_replica.cc:333] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb: stopping tablet replica
I20260430 08:01:21.102731 21820 raft_consensus.cc:2243] T a2623323d0f840428d0194bc45b42a8a P fb3b88c84b80445f96a8debdb48b5fdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:21.102761 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 21297
I20260430 08:01:21.103973 21818 ts_tablet_manager.cc:1916] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:21.104928 21819 tablet_replica.cc:333] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: stopping tablet replica
I20260430 08:01:21.105159 21819 raft_consensus.cc:2243] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:21.105326 21819 raft_consensus.cc:2272] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:21.106112 21818 ts_tablet_manager.cc:1929] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:21.106175 21819 ts_tablet_manager.cc:1916] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:21.106202 21818 log.cc:1199] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/a2623323d0f840428d0194bc45b42a8a
I20260430 08:01:21.106492 21818 ts_tablet_manager.cc:1950] T a2623323d0f840428d0194bc45b42a8a P 7183343b3df947c6817050cb5ae11661: Deleting consensus metadata
I20260430 08:01:21.107214 21218 catalog_manager.cc:5002] TS 7183343b3df947c6817050cb5ae11661 (127.13.94.131:33273): tablet a2623323d0f840428d0194bc45b42a8a (table test-table [id=3d30e08695ab4e939f720ae5da14abc9]) successfully deleted
I20260430 08:01:21.109146 21819 ts_tablet_manager.cc:1929] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:21.109231 21819 log.cc:1199] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/a2623323d0f840428d0194bc45b42a8a
I20260430 08:01:21.109539 21819 ts_tablet_manager.cc:1950] T a2623323d0f840428d0194bc45b42a8a P 42439d79f4494fef9d7096a3e096a023: Deleting consensus metadata
I20260430 08:01:21.110333 21217 catalog_manager.cc:5002] TS 42439d79f4494fef9d7096a3e096a023 (127.13.94.130:34387): tablet a2623323d0f840428d0194bc45b42a8a (table test-table [id=3d30e08695ab4e939f720ae5da14abc9]) successfully deleted
I20260430 08:01:21.111748 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 21433
I20260430 08:01:21.112501 21791 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:21.111974 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:36283 (local address 127.0.0.1:34828)
0430 08:01:21.112344 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:21.112399 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.129:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
W20260430 08:01:21.112900 21215 catalog_manager.cc:4729] TS fb3b88c84b80445f96a8debdb48b5fdb (127.13.94.129:36283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2623323d0f840428d0194bc45b42a8a: Network error: Client connection negotiation failed: client connection to 127.13.94.129:36283: connect: Connection refused (error 111)
I20260430 08:01:21.118530 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 21569
I20260430 08:01:21.124786 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 20907
I20260430 08:01:21.131085 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 21018
2026-04-30T08:01:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9311 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:01:21 dist-test-slave-9bxp krb5kdc[21823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:01:21 dist-test-slave-9bxp krb5kdc[21823](info): set up 2 sockets
Apr 30 08:01:21 dist-test-slave-9bxp krb5kdc[21823](info): commencing operation
krb5kdc: starting...
W20260430 08:01:23.145877 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.990s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:01:23 dist-test-slave-9bxp krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:01:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:23.298556 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:45879
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:45879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:23.406302 21839 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:23.406582 21839 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:23.406641 21839 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:23.411304 21839 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:23.411419 21839 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260430 08:01:23.411439 21839 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:23.411463 21839 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:23.411489 21839 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:23.411557 21839 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:23.417152 21839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:45879
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:45879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:23.418566 21839 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:23.419679 21839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260430 08:01:23.426620 21839 server_base.cc:1061] running on GCE node
W20260430 08:01:23.426674 21844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:23.426681 21847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:23.426716 21845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:23.427597 21839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:23.428637 21839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:23.429886 21839 hybrid_clock.cc:648] HybridClock initialized: now 1777536083429858 us; error 42 us; skew 500 ppm
Apr 30 08:01:23 dist-test-slave-9bxp krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536083, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:23.433017 21839 init.cc:377] Logged in from keytab as oryx/127.13.94.190@KRBTEST.COM (short username oryx)
I20260430 08:01:23.434285 21839 webserver.cc:492] Webserver started at http://127.13.94.190:42539/ using document root <none> and password file <none>
I20260430 08:01:23.434943 21839 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:23.435026 21839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:23.435261 21839 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:23.437273 21839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "fe04a193dc96440f8486edec4dca94c1"
format_stamp: "Formatted at 2026-04-30 08:01:23 on dist-test-slave-9bxp"
server_key: "e18815cd67091026ddecccd5e0381353"
server_key_iv: "fade8302bbf2f86ee0cbf0b0db573745"
server_key_version: "encryptionkey@0"
I20260430 08:01:23.437866 21839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "fe04a193dc96440f8486edec4dca94c1"
format_stamp: "Formatted at 2026-04-30 08:01:23 on dist-test-slave-9bxp"
server_key: "e18815cd67091026ddecccd5e0381353"
server_key_iv: "fade8302bbf2f86ee0cbf0b0db573745"
server_key_version: "encryptionkey@0"
I20260430 08:01:23.441867 21839 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260430 08:01:23.444670 21854 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:23.446059 21839 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260430 08:01:23.446272 21839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "fe04a193dc96440f8486edec4dca94c1"
format_stamp: "Formatted at 2026-04-30 08:01:23 on dist-test-slave-9bxp"
server_key: "e18815cd67091026ddecccd5e0381353"
server_key_iv: "fade8302bbf2f86ee0cbf0b0db573745"
server_key_version: "encryptionkey@0"
I20260430 08:01:23.446417 21839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:23.456833 21839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:23.463680 21839 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:23.463917 21839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:23.473024 21839 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:45879
I20260430 08:01:23.473354 21916 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:45879 every 8 connection(s)
I20260430 08:01:23.474172 21839 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:23.475315 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 21839
I20260430 08:01:23.475433 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:23.475708 13690 external_mini_cluster.cc:1468] Setting key cba23fe74d233a0cf7c6e6ffca123979
I20260430 08:01:23.478075 21917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Apr 30 08:01:23 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:23.483676 21917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1: Bootstrap starting.
I20260430 08:01:23.486318 21917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:23.487102 21917 log.cc:826] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:23.489017 21920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:23.477157 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.13.94.190:45879)
0430 08:01:23.477636 (+ 479us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:23.477646 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:23.477681 (+ 35us) server_negotiation.cc:408] Connection header received
0430 08:01:23.478414 (+ 733us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:23.478434 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:23.478731 (+ 297us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:23.479119 (+ 388us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:23.479998 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.480847 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:23.481553 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.481854 (+ 301us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:23.484326 (+ 2472us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:23.484382 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:23.484395 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:23.484425 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:23.486321 (+ 1896us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:23.486799 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:23.486804 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:23.486808 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:23.486880 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:23.487139 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:23.487143 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:23.487145 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:23.487532 (+ 387us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:23.487737 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:23.488063 (+ 326us) server_negotiation.cc:300] Negotiation successful
0430 08:01:23.488311 (+ 248us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":147,"threads_started":1}
I20260430 08:01:23.489338 21917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1: No bootstrap required, opened a new log
I20260430 08:01:23.492347 21917 raft_consensus.cc:359] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe04a193dc96440f8486edec4dca94c1" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45879 } }
I20260430 08:01:23.492575 21917 raft_consensus.cc:385] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:23.492666 21917 raft_consensus.cc:740] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe04a193dc96440f8486edec4dca94c1, State: Initialized, Role: FOLLOWER
I20260430 08:01:23.493122 21917 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe04a193dc96440f8486edec4dca94c1" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45879 } }
I20260430 08:01:23.493273 21917 raft_consensus.cc:399] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:23.493355 21917 raft_consensus.cc:493] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:23.493475 21917 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:23.494656 21917 raft_consensus.cc:515] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe04a193dc96440f8486edec4dca94c1" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45879 } }
I20260430 08:01:23.495205 21917 leader_election.cc:304] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe04a193dc96440f8486edec4dca94c1; no voters:
I20260430 08:01:23.495621 21917 leader_election.cc:290] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:23.495708 21922 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:23.495910 21922 raft_consensus.cc:697] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [term 1 LEADER]: Becoming Leader. State: Replica: fe04a193dc96440f8486edec4dca94c1, State: Running, Role: LEADER
I20260430 08:01:23.496295 21922 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe04a193dc96440f8486edec4dca94c1" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45879 } }
I20260430 08:01:23.497009 21917 sys_catalog.cc:565] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:23.498333 21924 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe04a193dc96440f8486edec4dca94c1. Latest consensus state: current_term: 1 leader_uuid: "fe04a193dc96440f8486edec4dca94c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe04a193dc96440f8486edec4dca94c1" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45879 } } }
I20260430 08:01:23.498478 21924 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:23.498760 21923 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe04a193dc96440f8486edec4dca94c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe04a193dc96440f8486edec4dca94c1" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 45879 } } }
I20260430 08:01:23.498970 21923 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:23.499006 21931 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:23.502507 21931 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:23.509220 21931 catalog_manager.cc:1357] Generated new cluster ID: 91092256f42046f796e5219c951fb38a
I20260430 08:01:23.509413 21931 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:23.521549 21931 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:23.522647 21931 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:23.533912 21931 catalog_manager.cc:6044] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1: Generated new TSK 0
I20260430 08:01:23.534698 21931 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:23.601835 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:45879
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:23.708851 21945 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:23.709214 21945 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:23.709347 21945 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:23.713738 21945 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:23.713805 21945 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:23.713896 21945 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 08:01:23.713964 21945 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:01:23.718461 21945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.94.190:45879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:23.719653 21945 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:23.720512 21945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:23.726683 21950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:23.726759 21951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:23.726981 21945 server_base.cc:1061] running on GCE node
W20260430 08:01:23.727011 21953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:23.727459 21945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:23.727952 21945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:23.729128 21945 hybrid_clock.cc:648] HybridClock initialized: now 1777536083729104 us; error 35 us; skew 500 ppm
Apr 30 08:01:23 dist-test-slave-9bxp krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536083, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:23.731740 21945 init.cc:377] Logged in from keytab as oryx/127.13.94.129@KRBTEST.COM (short username oryx)
I20260430 08:01:23.732859 21945 webserver.cc:492] Webserver started at http://127.13.94.129:33703/ using document root <none> and password file <none>
I20260430 08:01:23.733423 21945 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:23.733501 21945 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:23.733703 21945 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:23.735401 21945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "4bfb7409eeab4884a492e47bb318e059"
format_stamp: "Formatted at 2026-04-30 08:01:23 on dist-test-slave-9bxp"
server_key: "8489ab236d74859c66f3fc91b17a1fe8"
server_key_iv: "643694fae5b69a4835f78aa1a8c06d19"
server_key_version: "encryptionkey@0"
I20260430 08:01:23.735905 21945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "4bfb7409eeab4884a492e47bb318e059"
format_stamp: "Formatted at 2026-04-30 08:01:23 on dist-test-slave-9bxp"
server_key: "8489ab236d74859c66f3fc91b17a1fe8"
server_key_iv: "643694fae5b69a4835f78aa1a8c06d19"
server_key_version: "encryptionkey@0"
I20260430 08:01:23.739326 21945 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260430 08:01:23.741529 21960 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:23.742560 21945 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260430 08:01:23.742695 21945 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "4bfb7409eeab4884a492e47bb318e059"
format_stamp: "Formatted at 2026-04-30 08:01:23 on dist-test-slave-9bxp"
server_key: "8489ab236d74859c66f3fc91b17a1fe8"
server_key_iv: "643694fae5b69a4835f78aa1a8c06d19"
server_key_version: "encryptionkey@0"
I20260430 08:01:23.742810 21945 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:23.777004 21945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:23.780073 21945 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:23.780287 21945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:23.781936 21945 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:23.782020 21945 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:23.782085 21945 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:23.782136 21945 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 08:01:23 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536083, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:23.794543 21945 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:45127
I20260430 08:01:23.794582 22075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:45127 every 8 connection(s)
I20260430 08:01:23.794864 21920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:23.782890 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:50375 (local address 127.13.94.190:45879)
0430 08:01:23.783029 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:23.783033 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:23.783894 (+ 861us) server_negotiation.cc:408] Connection header received
0430 08:01:23.784910 (+ 1016us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:23.784917 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:23.784986 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:23.785111 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:23.786480 (+ 1369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.786998 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:23.787671 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.787873 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:23.790548 (+ 2675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:23.790578 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:23.790585 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:23.790625 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:23.792467 (+ 1842us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:23.793183 (+ 716us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:23.793186 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:23.793187 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:23.793236 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:23.793591 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:23.793594 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:23.793596 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:23.793804 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:23.793893 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:23.794542 (+ 649us) server_negotiation.cc:300] Negotiation successful
0430 08:01:23.794684 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 08:01:23.795531 21969 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:23.783230 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45879 (local address 127.13.94.129:50375)
0430 08:01:23.783740 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:23.783777 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:23.784598 (+ 821us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:23.785328 (+ 730us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:23.785342 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:23.785782 (+ 440us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:23.786317 (+ 535us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:23.786329 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.787141 (+ 812us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:23.787144 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:23.787542 (+ 398us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:23.787548 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.787776 (+ 228us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:23.788464 (+ 688us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:23.788493 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:23.790395 (+ 1902us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:23.792632 (+ 2237us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:23.792642 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:23.792660 (+ 18us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:23.793081 (+ 421us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:23.793353 (+ 272us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:23.793356 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:23.793359 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:23.793475 (+ 116us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:23.793926 (+ 451us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:23.793932 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:23.794239 (+ 307us) client_negotiation.cc:770] Sending connection context
0430 08:01:23.794483 (+ 244us) client_negotiation.cc:241] Negotiation successful
0430 08:01:23.794731 (+ 248us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":155,"threads_started":1}
I20260430 08:01:23.795688 21945 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:01:23.797225 22076 heartbeater.cc:344] Connected to a master server at 127.13.94.190:45879
I20260430 08:01:23.797389 22076 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:23.798017 22076 heartbeater.cc:507] Master 127.13.94.190:45879 requested a full tablet report, sending...
I20260430 08:01:23.798309 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 21945
I20260430 08:01:23.798444 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:01:23.798791 13690 external_mini_cluster.cc:1468] Setting key aea38109475eafb64cd9d6bb9b5035c2
I20260430 08:01:23.799849 21871 ts_manager.cc:194] Registered new tserver with Master: 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127)
I20260430 08:01:23.801394 21871 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:50375
I20260430 08:01:23.807740 22082 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:23.799809 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45879 (local address 127.13.94.129:51059)
0430 08:01:23.800383 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:23.800396 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:23.800538 (+ 142us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:23.800920 (+ 382us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:23.800922 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:23.801119 (+ 197us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:23.801334 (+ 215us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:23.801339 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.802248 (+ 909us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:23.802251 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:23.802869 (+ 618us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:23.802881 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.803024 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:23.803589 (+ 565us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:23.803614 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:23.803916 (+ 302us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:23.806258 (+ 2342us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:23.806261 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:23.806263 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:23.806459 (+ 196us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:23.806781 (+ 322us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:23.806784 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:23.806785 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:23.806834 (+ 49us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:23.807317 (+ 483us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:23.807320 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:23.807399 (+ 79us) client_negotiation.cc:770] Sending connection context
0430 08:01:23.807493 (+ 94us) client_negotiation.cc:241] Negotiation successful
0430 08:01:23.807599 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":101,"threads_started":1}
I20260430 08:01:23.807890 21920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:23.799815 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:51059 (local address 127.13.94.190:45879)
0430 08:01:23.799990 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:23.799993 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:23.800486 (+ 493us) server_negotiation.cc:408] Connection header received
0430 08:01:23.800634 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:23.800638 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:23.800734 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:23.800814 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:23.801464 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.802138 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:23.803016 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:23.803204 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:23.804057 (+ 853us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:23.804078 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:23.804084 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:23.804115 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:23.806009 (+ 1894us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:23.806574 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:23.806588 (+ 14us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:23.806597 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:23.806662 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:23.806939 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:23.806945 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:23.806948 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:23.807187 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:23.807308 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:23.807581 (+ 273us) server_negotiation.cc:300] Negotiation successful
0430 08:01:23.807715 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
WARNING: no policy specified for oryx/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:23.852046 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:45879
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:23.958703 22087 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:23.958935 22087 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:23.958994 22087 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:23.962414 22087 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:23.962488 22087 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:23.962541 22087 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 08:01:23.962584 22087 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:01:23.967170 22087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.94.190:45879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:23.968437 22087 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:23.969331 22087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:23.975759 22093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:23.975759 22092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:23.975775 22095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:23.976089 22087 server_base.cc:1061] running on GCE node
I20260430 08:01:23.976580 22087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:23.977180 22087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:23.978365 22087 hybrid_clock.cc:648] HybridClock initialized: now 1777536083978336 us; error 40 us; skew 500 ppm
Apr 30 08:01:23 dist-test-slave-9bxp krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536083, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:23.981053 22087 init.cc:377] Logged in from keytab as oryx/127.13.94.130@KRBTEST.COM (short username oryx)
I20260430 08:01:23.982223 22087 webserver.cc:492] Webserver started at http://127.13.94.130:38231/ using document root <none> and password file <none>
I20260430 08:01:23.982810 22087 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:23.982887 22087 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:23.983104 22087 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:23.984906 22087 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "7792d82ea35a4105a892c66260f1b546"
format_stamp: "Formatted at 2026-04-30 08:01:23 on dist-test-slave-9bxp"
server_key: "e4d65f3049627f0b297b977e3757a945"
server_key_iv: "01105723b109fc14148afb1654c53175"
server_key_version: "encryptionkey@0"
I20260430 08:01:23.985441 22087 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "7792d82ea35a4105a892c66260f1b546"
format_stamp: "Formatted at 2026-04-30 08:01:23 on dist-test-slave-9bxp"
server_key: "e4d65f3049627f0b297b977e3757a945"
server_key_iv: "01105723b109fc14148afb1654c53175"
server_key_version: "encryptionkey@0"
I20260430 08:01:23.989131 22087 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:01:23.991457 22102 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:23.992491 22087 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:23.992617 22087 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "7792d82ea35a4105a892c66260f1b546"
format_stamp: "Formatted at 2026-04-30 08:01:23 on dist-test-slave-9bxp"
server_key: "e4d65f3049627f0b297b977e3757a945"
server_key_iv: "01105723b109fc14148afb1654c53175"
server_key_version: "encryptionkey@0"
I20260430 08:01:23.992774 22087 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:24.014070 22087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:24.017066 22087 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:24.017287 22087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:24.018877 22087 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:24.018956 22087 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:24.019022 22087 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:24.019063 22087 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 08:01:24 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536083, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:24.031744 22087 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:36179
I20260430 08:01:24.031917 22217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:36179 every 8 connection(s)
I20260430 08:01:24.032665 21920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.019553 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:55957 (local address 127.13.94.190:45879)
0430 08:01:24.019713 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.019718 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.020535 (+ 817us) server_negotiation.cc:408] Connection header received
0430 08:01:24.021451 (+ 916us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.021455 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.021523 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.021636 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:24.023483 (+ 1847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.024347 (+ 864us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.025276 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.025538 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.028339 (+ 2801us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:24.028367 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:24.028371 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:24.028416 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:24.030478 (+ 2062us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.030968 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.030974 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.030978 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.031048 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.031427 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.031431 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.031433 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.031661 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:24.031807 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.032433 (+ 626us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.032544 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260430 08:01:24.033097 22087 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:01:24.033517 22111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.019907 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45879 (local address 127.13.94.130:55957)
0430 08:01:24.020377 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.020418 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.021245 (+ 827us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.021784 (+ 539us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.021797 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.022534 (+ 737us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:24.023276 (+ 742us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.023290 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.024521 (+ 1231us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.024525 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.025155 (+ 630us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.025164 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.025395 (+ 231us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.026147 (+ 752us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:24.026168 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:24.028145 (+ 1977us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:24.030617 (+ 2472us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.030622 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.030631 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.030861 (+ 230us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.031178 (+ 317us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.031181 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.031183 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.031275 (+ 92us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.031825 (+ 550us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:24.031833 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:24.032142 (+ 309us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.032382 (+ 240us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.032626 (+ 244us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":139,"threads_started":1}
I20260430 08:01:24.035193 22218 heartbeater.cc:344] Connected to a master server at 127.13.94.190:45879
I20260430 08:01:24.035385 22218 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:24.035928 22218 heartbeater.cc:507] Master 127.13.94.190:45879 requested a full tablet report, sending...
I20260430 08:01:24.037037 21870 ts_manager.cc:194] Registered new tserver with Master: 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179)
I20260430 08:01:24.037778 21870 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:55957
I20260430 08:01:24.038682 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 22087
I20260430 08:01:24.038800 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:01:24.039095 13690 external_mini_cluster.cc:1468] Setting key cefc751a634855210351bd541d7d836f
I20260430 08:01:24.045153 21920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.037657 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:41023 (local address 127.13.94.190:45879)
0430 08:01:24.037795 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.037802 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.038063 (+ 261us) server_negotiation.cc:408] Connection header received
0430 08:01:24.038211 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.038214 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.038270 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.038335 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:24.039074 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.039816 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.040596 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.040837 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.041617 (+ 780us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:24.041638 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:24.041644 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:24.041678 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:24.043353 (+ 1675us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.043930 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.043934 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.043936 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.043997 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.044299 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.044302 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.044303 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.044470 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:24.044566 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.044902 (+ 336us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.045009 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260430 08:01:24.045305 22224 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.037533 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45879 (local address 127.13.94.130:41023)
0430 08:01:24.037962 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.037978 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.038090 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.038378 (+ 288us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.038382 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.038628 (+ 246us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:24.038937 (+ 309us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.038948 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.039943 (+ 995us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.039947 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.040479 (+ 532us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.040487 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.040582 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.041187 (+ 605us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:24.041200 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:24.041489 (+ 289us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:24.043487 (+ 1998us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.043493 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.043499 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.043768 (+ 269us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.044125 (+ 357us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.044131 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.044135 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.044203 (+ 68us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.044576 (+ 373us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:24.044582 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:24.044689 (+ 107us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.044847 (+ 158us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.045005 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab.
I20260430 08:01:24.092228 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:45879
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:24.195809 22229 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:24.196043 22229 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:24.196112 22229 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:24.199766 22229 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:24.199836 22229 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:24.199890 22229 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260430 08:01:24.199941 22229 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:01:24.204317 22229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.94.190:45879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:24.205403 22229 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:24.206195 22229 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:24.212899 22237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:24.212952 22235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:24.213039 22234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:24.212947 22229 server_base.cc:1061] running on GCE node
I20260430 08:01:24.213629 22229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:24.214233 22229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:24.215394 22229 hybrid_clock.cc:648] HybridClock initialized: now 1777536084215369 us; error 38 us; skew 500 ppm
Apr 30 08:01:24 dist-test-slave-9bxp krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536084, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:24.218250 22229 init.cc:377] Logged in from keytab as oryx/127.13.94.131@KRBTEST.COM (short username oryx)
I20260430 08:01:24.219360 22229 webserver.cc:492] Webserver started at http://127.13.94.131:37449/ using document root <none> and password file <none>
I20260430 08:01:24.219935 22229 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:24.220010 22229 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:24.220224 22229 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:24.222110 22229 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "6f661fe8f5e241acbf8b5482b32be069"
format_stamp: "Formatted at 2026-04-30 08:01:24 on dist-test-slave-9bxp"
server_key: "1c20058d178926319f52e2cefab20a43"
server_key_iv: "ec2cb2e9e7f6d018a28790583a52340d"
server_key_version: "encryptionkey@0"
I20260430 08:01:24.222601 22229 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "6f661fe8f5e241acbf8b5482b32be069"
format_stamp: "Formatted at 2026-04-30 08:01:24 on dist-test-slave-9bxp"
server_key: "1c20058d178926319f52e2cefab20a43"
server_key_iv: "ec2cb2e9e7f6d018a28790583a52340d"
server_key_version: "encryptionkey@0"
I20260430 08:01:24.225907 22229 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260430 08:01:24.228084 22244 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:24.229149 22229 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:24.229278 22229 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "6f661fe8f5e241acbf8b5482b32be069"
format_stamp: "Formatted at 2026-04-30 08:01:24 on dist-test-slave-9bxp"
server_key: "1c20058d178926319f52e2cefab20a43"
server_key_iv: "ec2cb2e9e7f6d018a28790583a52340d"
server_key_version: "encryptionkey@0"
I20260430 08:01:24.229385 22229 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:24.239881 22229 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:24.242957 22229 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:24.243182 22229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:24.244778 22229 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:24.244853 22229 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:24.244912 22229 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:24.244946 22229 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 30 08:01:24 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536084, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:24.257001 22229 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:42691
I20260430 08:01:24.257084 22359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:42691 every 8 connection(s)
I20260430 08:01:24.258198 22229 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:01:24.258217 21920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.245424 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:42439 (local address 127.13.94.190:45879)
0430 08:01:24.245582 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.245587 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.246428 (+ 841us) server_negotiation.cc:408] Connection header received
0430 08:01:24.247618 (+ 1190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.247622 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.247692 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.247774 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:24.249613 (+ 1839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.250191 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.251103 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.251342 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.254216 (+ 2874us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:24.254245 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:24.254251 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:24.254292 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:24.255989 (+ 1697us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.256594 (+ 605us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.256598 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.256600 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.256657 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.257090 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.257094 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.257096 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.257299 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:24.257402 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.257902 (+ 500us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.258029 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260430 08:01:24.258487 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 22229
I20260430 08:01:24.258606 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:01:24.258886 13690 external_mini_cluster.cc:1468] Setting key 360a2fa73da30c1bb578c8e4d0982069
I20260430 08:01:24.258905 22253 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.245750 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45879 (local address 127.13.94.131:42439)
0430 08:01:24.246241 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.246293 (+ 52us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.247373 (+ 1080us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.247999 (+ 626us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.248015 (+ 16us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.248644 (+ 629us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:24.249409 (+ 765us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.249429 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.250310 (+ 881us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.250318 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.250962 (+ 644us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.250974 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.251212 (+ 238us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.251909 (+ 697us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:24.251940 (+ 31us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:24.254017 (+ 2077us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:24.256109 (+ 2092us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.256116 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.256128 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.256477 (+ 349us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.256792 (+ 315us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.256799 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.256804 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.256951 (+ 147us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.257412 (+ 461us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:24.257417 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:24.257631 (+ 214us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.257810 (+ 179us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.258039 (+ 229us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":103,"threads_started":1}
I20260430 08:01:24.260305 22360 heartbeater.cc:344] Connected to a master server at 127.13.94.190:45879
I20260430 08:01:24.260517 22360 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:24.261135 22360 heartbeater.cc:507] Master 127.13.94.190:45879 requested a full tablet report, sending...
I20260430 08:01:24.262274 21871 ts_manager.cc:194] Registered new tserver with Master: 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691)
I20260430 08:01:24.262939 21871 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:42439
I20260430 08:01:24.269145 22366 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.262775 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45879 (local address 127.13.94.131:48913)
0430 08:01:24.263052 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.263069 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.263173 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.263594 (+ 421us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.263597 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.263780 (+ 183us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:24.264016 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.264022 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.265006 (+ 984us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.265009 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.265383 (+ 374us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.265389 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.265482 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.266043 (+ 561us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:24.266056 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:24.266354 (+ 298us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:24.267906 (+ 1552us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.267909 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.267910 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.268075 (+ 165us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.268314 (+ 239us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.268316 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.268317 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.268358 (+ 41us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.268640 (+ 282us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:24.268642 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:24.268789 (+ 147us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.268904 (+ 115us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.269001 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260430 08:01:24.269171 21920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.263255 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:48913 (local address 127.13.94.190:45879)
0430 08:01:24.263382 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.263385 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.263395 (+ 10us) server_negotiation.cc:408] Connection header received
0430 08:01:24.263422 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.263425 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.263483 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.263562 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:24.264142 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.264892 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.265483 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.265637 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.266469 (+ 832us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:24.266484 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:24.266486 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:24.266506 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:24.267784 (+ 1278us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.268178 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.268181 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.268182 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.268226 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.268433 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.268435 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.268436 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.268563 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:24.268638 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.268963 (+ 325us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.269056 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 08:01:24.273072 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 30 08:01:24 dist-test-slave-9bxp krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 30 08:01:24 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:24 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
W20260430 08:01:24.297477 21920 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:24.290503 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.13.94.190:45879)
0430 08:01:24.290643 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.290646 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.290659 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:24.290705 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.290708 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.290754 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.290888 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:24.291890 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.292388 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.293258 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.293454 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.297290 (+ 3836us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39706: BlockingRecv error: recv got EOF from 127.0.0.1:39706 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Apr 30 08:01:24 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:24.307977 21920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.299289 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.13.94.190:45879)
0430 08:01:24.299476 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.299480 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.299617 (+ 137us) server_negotiation.cc:408] Connection header received
0430 08:01:24.299795 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.299798 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.299847 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.299948 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:24.300749 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.301255 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.301900 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.302110 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.304486 (+ 2376us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:24.304508 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:24.304510 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:24.304541 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:24.306141 (+ 1600us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.306602 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.306606 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.306607 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.306662 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.306993 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.306997 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.306998 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.307172 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:24.307292 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.307646 (+ 354us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.307768 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260430 08:01:24.311363 21871 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:01:24.313985 21871 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:01:24.327970 22383 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.322784 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:42691 (local address 127.0.0.1:42634)
0430 08:01:24.323462 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.323501 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.323644 (+ 143us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.324464 (+ 820us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.324471 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.324493 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:24.324832 (+ 339us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.324841 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.326484 (+ 1643us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.326488 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.327453 (+ 965us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.327462 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.327574 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.327636 (+ 62us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.327733 (+ 97us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.327806 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":85,"threads_started":1}
I20260430 08:01:24.328611 22384 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.322995 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.13.94.131:42691)
0430 08:01:24.323979 (+ 984us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.323985 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.324006 (+ 21us) server_negotiation.cc:408] Connection header received
0430 08:01:24.324097 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.324103 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.324293 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.324992 (+ 699us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:24.325108 (+ 116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.326318 (+ 1210us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.327605 (+ 1287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.328169 (+ 564us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.328292 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.328363 (+ 71us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.328453 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":130,"threads_started":1}
I20260430 08:01:24.328791 22385 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.323038 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:45127 (local address 127.0.0.1:34850)
0430 08:01:24.324001 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.324014 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.324170 (+ 156us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.324813 (+ 643us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.324816 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.324853 (+ 37us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:24.325142 (+ 289us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.325149 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.326461 (+ 1312us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.326755 (+ 294us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.327509 (+ 754us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.327541 (+ 32us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.328427 (+ 886us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.328441 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.328551 (+ 110us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.328603 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"spinlock_wait_cycles":6528,"thread_start_us":103,"threads_started":1}
I20260430 08:01:24.328969 22386 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.323510 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.13.94.129:45127)
0430 08:01:24.323870 (+ 360us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.323877 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.324092 (+ 215us) server_negotiation.cc:408] Connection header received
0430 08:01:24.324339 (+ 247us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.324344 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.324515 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.324604 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:24.325279 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.326340 (+ 1061us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.327696 (+ 1356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.328265 (+ 569us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.328347 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.328654 (+ 307us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.328805 (+ 151us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":77,"threads_started":1}
I20260430 08:01:24.329051 22382 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.323469 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:36179 (local address 127.0.0.1:37040)
0430 08:01:24.324011 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.324025 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.324106 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.324813 (+ 707us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.324816 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.324852 (+ 36us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:24.325142 (+ 290us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.325149 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.327746 (+ 2597us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.327750 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.328656 (+ 906us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.328667 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.328806 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.328819 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.328870 (+ 51us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.328920 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"spinlock_wait_cycles":10112,"thread_start_us":79,"threads_started":1}
I20260430 08:01:24.330133 22387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.323632 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.13.94.130:36179)
0430 08:01:24.324414 (+ 782us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.324422 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.324442 (+ 20us) server_negotiation.cc:408] Connection header received
0430 08:01:24.324514 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.324518 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.324664 (+ 146us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.324823 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:24.326485 (+ 1662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.327600 (+ 1115us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.329237 (+ 1637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.329743 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.329845 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.329909 (+ 64us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.329988 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":120,"threads_started":1}
I20260430 08:01:24.331717 22294 tablet_service.cc:1511] Processing CreateTablet for tablet 17fcdbef2a5c447f9a459c967d639a80 (DEFAULT_TABLE table=test-table [id=4216d8f68d744c8494b2677585551c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:24.331748 22010 tablet_service.cc:1511] Processing CreateTablet for tablet 17fcdbef2a5c447f9a459c967d639a80 (DEFAULT_TABLE table=test-table [id=4216d8f68d744c8494b2677585551c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:24.332612 22294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fcdbef2a5c447f9a459c967d639a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:24.332625 22010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fcdbef2a5c447f9a459c967d639a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:24.332466 22152 tablet_service.cc:1511] Processing CreateTablet for tablet 17fcdbef2a5c447f9a459c967d639a80 (DEFAULT_TABLE table=test-table [id=4216d8f68d744c8494b2677585551c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:24.333283 22152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17fcdbef2a5c447f9a459c967d639a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:24.338220 22388 tablet_bootstrap.cc:492] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: Bootstrap starting.
I20260430 08:01:24.339277 22389 tablet_bootstrap.cc:492] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: Bootstrap starting.
I20260430 08:01:24.339277 22390 tablet_bootstrap.cc:492] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: Bootstrap starting.
I20260430 08:01:24.341553 22388 tablet_bootstrap.cc:654] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:24.341558 22390 tablet_bootstrap.cc:654] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:24.341558 22389 tablet_bootstrap.cc:654] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:24.342523 22389 log.cc:826] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:24.342566 22388 log.cc:826] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:24.342690 22390 log.cc:826] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:24.344322 22389 tablet_bootstrap.cc:492] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: No bootstrap required, opened a new log
I20260430 08:01:24.344322 22388 tablet_bootstrap.cc:492] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: No bootstrap required, opened a new log
I20260430 08:01:24.344535 22389 ts_tablet_manager.cc:1403] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260430 08:01:24.344549 22388 ts_tablet_manager.cc:1403] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260430 08:01:24.345734 22390 tablet_bootstrap.cc:492] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: No bootstrap required, opened a new log
I20260430 08:01:24.345933 22390 ts_tablet_manager.cc:1403] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260430 08:01:24.347231 22389 raft_consensus.cc:359] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } }
I20260430 08:01:24.347477 22389 raft_consensus.cc:385] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:24.347546 22389 raft_consensus.cc:740] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfb7409eeab4884a492e47bb318e059, State: Initialized, Role: FOLLOWER
I20260430 08:01:24.347719 22388 raft_consensus.cc:359] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } }
I20260430 08:01:24.347949 22388 raft_consensus.cc:385] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:24.348007 22388 raft_consensus.cc:740] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f661fe8f5e241acbf8b5482b32be069, State: Initialized, Role: FOLLOWER
I20260430 08:01:24.348040 22389 consensus_queue.cc:260] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } }
I20260430 08:01:24.348446 22388 consensus_queue.cc:260] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } }
I20260430 08:01:24.348910 22076 heartbeater.cc:499] Master 127.13.94.190:45879 was elected leader, sending a full tablet report...
I20260430 08:01:24.349018 22389 ts_tablet_manager.cc:1434] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260430 08:01:24.349171 22390 raft_consensus.cc:359] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } }
I20260430 08:01:24.349387 22388 ts_tablet_manager.cc:1434] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 08:01:24.349390 22390 raft_consensus.cc:385] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:24.349450 22390 raft_consensus.cc:740] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7792d82ea35a4105a892c66260f1b546, State: Initialized, Role: FOLLOWER
I20260430 08:01:24.349730 22360 heartbeater.cc:499] Master 127.13.94.190:45879 was elected leader, sending a full tablet report...
I20260430 08:01:24.349875 22390 consensus_queue.cc:260] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } }
I20260430 08:01:24.350625 22218 heartbeater.cc:499] Master 127.13.94.190:45879 was elected leader, sending a full tablet report...
I20260430 08:01:24.350767 22390 ts_tablet_manager.cc:1434] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 08:01:24.426245 22396 raft_consensus.cc:493] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:01:24.426470 22396 raft_consensus.cc:515] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } }
I20260430 08:01:24.427644 22396 leader_election.cc:290] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691), 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127)
I20260430 08:01:24.431634 22111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.427943 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:45127 (local address 127.13.94.130:47885)
0430 08:01:24.428098 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.428117 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.428223 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.428524 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.428527 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.428541 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:24.428854 (+ 313us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.428863 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.430315 (+ 1452us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.430319 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.431231 (+ 912us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.431236 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.431346 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.431364 (+ 18us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.431416 (+ 52us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.431467 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260430 08:01:24.431650 22397 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.427939 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:42691 (local address 127.13.94.130:45561)
0430 08:01:24.428417 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.428428 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.428537 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.428820 (+ 283us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.428823 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.428847 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:24.429115 (+ 268us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.429121 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.430258 (+ 1137us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.430265 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.431207 (+ 942us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.431218 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.431346 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.431364 (+ 18us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.431416 (+ 52us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.431467 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":59,"thread_start_us":137,"threads_started":1}
I20260430 08:01:24.432138 22386 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.428052 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:47885 (local address 127.13.94.129:45127)
0430 08:01:24.428208 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.428213 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.428231 (+ 18us) server_negotiation.cc:408] Connection header received
0430 08:01:24.428327 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.428332 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.428389 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.428484 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:24.428980 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.430199 (+ 1219us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.431354 (+ 1155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.431861 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.431936 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.431984 (+ 48us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.432038 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260430 08:01:24.432128 22384 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.428199 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:45561 (local address 127.13.94.131:42691)
0430 08:01:24.428385 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.428389 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.428533 (+ 144us) server_negotiation.cc:408] Connection header received
0430 08:01:24.428630 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.428633 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.428672 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.428813 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:24.429239 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.430111 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.431354 (+ 1243us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.431861 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.431898 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.431955 (+ 57us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.432012 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260430 08:01:24.432823 22308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fcdbef2a5c447f9a459c967d639a80" candidate_uuid: "7792d82ea35a4105a892c66260f1b546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f661fe8f5e241acbf8b5482b32be069" is_pre_election: true
I20260430 08:01:24.432818 22030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fcdbef2a5c447f9a459c967d639a80" candidate_uuid: "7792d82ea35a4105a892c66260f1b546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfb7409eeab4884a492e47bb318e059" is_pre_election: true
I20260430 08:01:24.433131 22030 raft_consensus.cc:2468] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7792d82ea35a4105a892c66260f1b546 in term 0.
I20260430 08:01:24.433159 22308 raft_consensus.cc:2468] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7792d82ea35a4105a892c66260f1b546 in term 0.
I20260430 08:01:24.433624 22105 leader_election.cc:304] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfb7409eeab4884a492e47bb318e059, 7792d82ea35a4105a892c66260f1b546; no voters:
I20260430 08:01:24.433909 22396 raft_consensus.cc:2804] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:01:24.433987 22396 raft_consensus.cc:493] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:01:24.434033 22396 raft_consensus.cc:3060] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:24.435001 22396 raft_consensus.cc:515] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } }
I20260430 08:01:24.435446 22396 leader_election.cc:290] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [CANDIDATE]: Term 1 election: Requested vote from peers 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691), 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127)
I20260430 08:01:24.435950 22308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fcdbef2a5c447f9a459c967d639a80" candidate_uuid: "7792d82ea35a4105a892c66260f1b546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f661fe8f5e241acbf8b5482b32be069"
I20260430 08:01:24.436021 22030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17fcdbef2a5c447f9a459c967d639a80" candidate_uuid: "7792d82ea35a4105a892c66260f1b546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfb7409eeab4884a492e47bb318e059"
I20260430 08:01:24.436074 22308 raft_consensus.cc:3060] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:24.436201 22030 raft_consensus.cc:3060] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:24.437247 22308 raft_consensus.cc:2468] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7792d82ea35a4105a892c66260f1b546 in term 1.
I20260430 08:01:24.437458 22030 raft_consensus.cc:2468] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7792d82ea35a4105a892c66260f1b546 in term 1.
I20260430 08:01:24.437642 22103 leader_election.cc:304] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f661fe8f5e241acbf8b5482b32be069, 7792d82ea35a4105a892c66260f1b546; no voters:
I20260430 08:01:24.437849 22396 raft_consensus.cc:2804] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:24.438112 22396 raft_consensus.cc:697] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 1 LEADER]: Becoming Leader. State: Replica: 7792d82ea35a4105a892c66260f1b546, State: Running, Role: LEADER
I20260430 08:01:24.438453 22396 consensus_queue.cc:237] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } }
I20260430 08:01:24.441882 21871 catalog_manager.cc:5671] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7792d82ea35a4105a892c66260f1b546 (127.13.94.130). New cstate: current_term: 1 leader_uuid: "7792d82ea35a4105a892c66260f1b546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } health_report { overall_health: HEALTHY } } }
I20260430 08:01:24.454213 22387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.451212 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.13.94.130:36179)
0430 08:01:24.451347 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.451351 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.451363 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:01:24.451463 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.451467 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.451528 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.451648 (+ 120us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:24.451989 (+ 341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.452486 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.453208 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.453353 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.453397 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:24.453762 (+ 365us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:24.453848 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.453970 (+ 122us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.454039 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260430 08:01:24.459661 22308 raft_consensus.cc:1275] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Refusing update from remote peer 7792d82ea35a4105a892c66260f1b546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:24.459659 22030 raft_consensus.cc:1275] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Refusing update from remote peer 7792d82ea35a4105a892c66260f1b546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:24.460357 22396 consensus_queue.cc:1048] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:24.460554 22398 consensus_queue.cc:1048] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:24.476465 21871 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:24.476639 21871 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:24.477368 22402 mvcc.cc:204] Tried to move back new op lower bound from 7280787801939746816 to 7280787801862995968. Current Snapshot: MvccSnapshot[applied={T|T < 7280787801939746816}]
I20260430 08:01:24.479455 21871 catalog_manager.cc:5958] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1: Sending DeleteTablet for 3 replicas of tablet 17fcdbef2a5c447f9a459c967d639a80
I20260430 08:01:24.480201 22294 tablet_service.cc:1558] Processing DeleteTablet for tablet 17fcdbef2a5c447f9a459c967d639a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:24 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42634
I20260430 08:01:24.480512 22152 tablet_service.cc:1558] Processing DeleteTablet for tablet 17fcdbef2a5c447f9a459c967d639a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:24 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:37040
I20260430 08:01:24.480911 22010 tablet_service.cc:1558] Processing DeleteTablet for tablet 17fcdbef2a5c447f9a459c967d639a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:24 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:34850
I20260430 08:01:24.480964 22411 tablet_replica.cc:333] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: stopping tablet replica
I20260430 08:01:24.481259 22411 raft_consensus.cc:2243] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:24.481599 22411 raft_consensus.cc:2272] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:24.481889 21869 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:01:24.482371 21869 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:01:24.482919 22412 tablet_replica.cc:333] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: stopping tablet replica
I20260430 08:01:24.483176 22412 raft_consensus.cc:2243] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:01:24.483506 22412 raft_consensus.cc:2272] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:24.483846 22413 tablet_replica.cc:333] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: stopping tablet replica
I20260430 08:01:24.484066 22413 raft_consensus.cc:2243] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:24.484339 22413 raft_consensus.cc:2272] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:24.485586 22411 ts_tablet_manager.cc:1916] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:24.486562 22412 ts_tablet_manager.cc:1916] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:24.488046 22411 ts_tablet_manager.cc:1929] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:24.488140 22411 log.cc:1199] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/17fcdbef2a5c447f9a459c967d639a80
I20260430 08:01:24.488415 22411 ts_tablet_manager.cc:1950] T 17fcdbef2a5c447f9a459c967d639a80 P 6f661fe8f5e241acbf8b5482b32be069: Deleting consensus metadata
I20260430 08:01:24.488965 22010 tablet_service.cc:1511] Processing CreateTablet for tablet 6edf7a6ebd34478891ea4c5cf6d37239 (DEFAULT_TABLE table=test-table [id=c86764180a094c9b8a423fd8c1edb52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:24.489085 22152 tablet_service.cc:1511] Processing CreateTablet for tablet 6edf7a6ebd34478891ea4c5cf6d37239 (DEFAULT_TABLE table=test-table [id=c86764180a094c9b8a423fd8c1edb52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:24.489269 22010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edf7a6ebd34478891ea4c5cf6d37239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:24.489331 22152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edf7a6ebd34478891ea4c5cf6d37239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:24.491060 22412 ts_tablet_manager.cc:1929] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:24.492076 22389 tablet_bootstrap.cc:492] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: Bootstrap starting.
I20260430 08:01:24.493083 21855 catalog_manager.cc:5002] TS 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691): tablet 17fcdbef2a5c447f9a459c967d639a80 (table test-table [id=4216d8f68d744c8494b2677585551c52]) successfully deleted
I20260430 08:01:24.493211 22389 tablet_bootstrap.cc:654] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:24.493317 22412 log.cc:1199] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/17fcdbef2a5c447f9a459c967d639a80
I20260430 08:01:24.493621 22412 ts_tablet_manager.cc:1950] T 17fcdbef2a5c447f9a459c967d639a80 P 7792d82ea35a4105a892c66260f1b546: Deleting consensus metadata
I20260430 08:01:24.494001 22294 tablet_service.cc:1511] Processing CreateTablet for tablet 6edf7a6ebd34478891ea4c5cf6d37239 (DEFAULT_TABLE table=test-table [id=c86764180a094c9b8a423fd8c1edb52f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:24.494295 22294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edf7a6ebd34478891ea4c5cf6d37239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:24.494593 22413 ts_tablet_manager.cc:1916] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:24.494692 21855 catalog_manager.cc:5002] TS 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179): tablet 17fcdbef2a5c447f9a459c967d639a80 (table test-table [id=4216d8f68d744c8494b2677585551c52]) successfully deleted
I20260430 08:01:24.491930 22390 tablet_bootstrap.cc:492] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: Bootstrap starting.
I20260430 08:01:24.495910 22390 tablet_bootstrap.cc:654] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:24.496518 22388 tablet_bootstrap.cc:492] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: Bootstrap starting.
I20260430 08:01:24.497372 22413 ts_tablet_manager.cc:1929] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:24.497505 22413 log.cc:1199] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/17fcdbef2a5c447f9a459c967d639a80
I20260430 08:01:24.497597 22388 tablet_bootstrap.cc:654] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:24.497759 22389 tablet_bootstrap.cc:492] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: No bootstrap required, opened a new log
I20260430 08:01:24.497875 22389 ts_tablet_manager.cc:1403] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260430 08:01:24.498082 22413 ts_tablet_manager.cc:1950] T 17fcdbef2a5c447f9a459c967d639a80 P 4bfb7409eeab4884a492e47bb318e059: Deleting consensus metadata
I20260430 08:01:24.498486 22389 raft_consensus.cc:359] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:24.498607 22389 raft_consensus.cc:385] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:24.498646 22389 raft_consensus.cc:740] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfb7409eeab4884a492e47bb318e059, State: Initialized, Role: FOLLOWER
I20260430 08:01:24.499207 22388 tablet_bootstrap.cc:492] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: No bootstrap required, opened a new log
I20260430 08:01:24.499300 22388 ts_tablet_manager.cc:1403] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 08:01:24.499579 22389 consensus_queue.cc:260] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:24.499651 21857 catalog_manager.cc:5002] TS 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127): tablet 17fcdbef2a5c447f9a459c967d639a80 (table test-table [id=4216d8f68d744c8494b2677585551c52]) successfully deleted
I20260430 08:01:24.499774 22388 raft_consensus.cc:359] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:24.499894 22388 raft_consensus.cc:385] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:24.499935 22388 raft_consensus.cc:740] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f661fe8f5e241acbf8b5482b32be069, State: Initialized, Role: FOLLOWER
I20260430 08:01:24.499996 22389 ts_tablet_manager.cc:1434] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260430 08:01:24.500082 22388 consensus_queue.cc:260] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:24.500339 22388 ts_tablet_manager.cc:1434] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:01:24.501019 22390 tablet_bootstrap.cc:492] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: No bootstrap required, opened a new log
I20260430 08:01:24.501119 22390 ts_tablet_manager.cc:1403] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260430 08:01:24.501622 22390 raft_consensus.cc:359] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:24.501729 22390 raft_consensus.cc:385] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:24.501770 22390 raft_consensus.cc:740] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7792d82ea35a4105a892c66260f1b546, State: Initialized, Role: FOLLOWER
I20260430 08:01:24.501951 22390 consensus_queue.cc:260] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:24.502238 22390 ts_tablet_manager.cc:1434] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260430 08:01:24.510422 22361 tablet.cc:2404] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:01:24.534778 22219 tablet.cc:2404] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260430 08:01:24.547310 22077 tablet.cc:2404] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 08:01:24.862093 22395 raft_consensus.cc:493] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:01:24.862287 22395 raft_consensus.cc:515] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:24.863379 22395 leader_election.cc:290] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127), 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179)
I20260430 08:01:24.866583 22417 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.863654 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:36179 (local address 127.13.94.131:47241)
0430 08:01:24.864073 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.864086 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.864227 (+ 141us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.864507 (+ 280us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.864510 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.864529 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:24.864797 (+ 268us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.864803 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.865512 (+ 709us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.865517 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.866092 (+ 575us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.866098 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.866232 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.866249 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.866303 (+ 54us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.866357 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"spinlock_wait_cycles":26368,"thread_start_us":95,"threads_started":1}
I20260430 08:01:24.866590 22418 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.863654 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:45127 (local address 127.13.94.131:35169)
0430 08:01:24.864110 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.864122 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.864262 (+ 140us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.864538 (+ 276us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.864541 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.864582 (+ 41us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:24.864861 (+ 279us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.864867 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.865572 (+ 705us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.865579 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.866156 (+ 577us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.866163 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.866298 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.866312 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.866363 (+ 51us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.866410 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":38,"spinlock_wait_cycles":768,"thread_start_us":69,"threads_started":1}
I20260430 08:01:24.867007 22387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.863799 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:47241 (local address 127.13.94.130:36179)
0430 08:01:24.863945 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.863950 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.864173 (+ 223us) server_negotiation.cc:408] Connection header received
0430 08:01:24.864353 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.864360 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.864406 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.864495 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:24.864915 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.865399 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.866220 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.866702 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.866741 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.866807 (+ 66us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.866870 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260430 08:01:24.867013 22386 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.863783 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:35169 (local address 127.13.94.129:45127)
0430 08:01:24.863958 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.863963 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.864214 (+ 251us) server_negotiation.cc:408] Connection header received
0430 08:01:24.864381 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.864384 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.864439 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.864526 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:24.864998 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.865458 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.866282 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.866748 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.866787 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.866842 (+ 55us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.866897 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 08:01:24.867455 22030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6edf7a6ebd34478891ea4c5cf6d37239" candidate_uuid: "6f661fe8f5e241acbf8b5482b32be069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfb7409eeab4884a492e47bb318e059" is_pre_election: true
I20260430 08:01:24.867519 22172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6edf7a6ebd34478891ea4c5cf6d37239" candidate_uuid: "6f661fe8f5e241acbf8b5482b32be069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792d82ea35a4105a892c66260f1b546" is_pre_election: true
I20260430 08:01:24.867672 22030 raft_consensus.cc:2468] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f661fe8f5e241acbf8b5482b32be069 in term 0.
I20260430 08:01:24.867744 22172 raft_consensus.cc:2468] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f661fe8f5e241acbf8b5482b32be069 in term 0.
I20260430 08:01:24.868222 22245 leader_election.cc:304] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f661fe8f5e241acbf8b5482b32be069, 7792d82ea35a4105a892c66260f1b546; no voters:
I20260430 08:01:24.868484 22395 raft_consensus.cc:2804] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:01:24.868551 22395 raft_consensus.cc:493] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:01:24.868589 22395 raft_consensus.cc:3060] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:24.869568 22395 raft_consensus.cc:515] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:24.869874 22395 leader_election.cc:290] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [CANDIDATE]: Term 1 election: Requested vote from peers 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127), 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179)
I20260430 08:01:24.870271 22172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6edf7a6ebd34478891ea4c5cf6d37239" candidate_uuid: "6f661fe8f5e241acbf8b5482b32be069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792d82ea35a4105a892c66260f1b546"
I20260430 08:01:24.870270 22030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6edf7a6ebd34478891ea4c5cf6d37239" candidate_uuid: "6f661fe8f5e241acbf8b5482b32be069" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfb7409eeab4884a492e47bb318e059"
I20260430 08:01:24.870371 22172 raft_consensus.cc:3060] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:24.870405 22030 raft_consensus.cc:3060] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:24.871201 22030 raft_consensus.cc:2468] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f661fe8f5e241acbf8b5482b32be069 in term 1.
I20260430 08:01:24.871353 22172 raft_consensus.cc:2468] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f661fe8f5e241acbf8b5482b32be069 in term 1.
I20260430 08:01:24.871515 22247 leader_election.cc:304] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfb7409eeab4884a492e47bb318e059, 6f661fe8f5e241acbf8b5482b32be069; no voters:
I20260430 08:01:24.871706 22395 raft_consensus.cc:2804] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:24.871868 22395 raft_consensus.cc:697] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 LEADER]: Becoming Leader. State: Replica: 6f661fe8f5e241acbf8b5482b32be069, State: Running, Role: LEADER
I20260430 08:01:24.872114 22395 consensus_queue.cc:237] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:24.874425 21871 catalog_manager.cc:5671] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131). New cstate: current_term: 1 leader_uuid: "6f661fe8f5e241acbf8b5482b32be069" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } health_report { overall_health: HEALTHY } } }
Apr 30 08:01:24 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536083, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.190@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:24.998965 22426 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.990619 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:45879 (local address 127.0.0.1:39714)
0430 08:01:24.991081 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:24.991097 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:24.991256 (+ 159us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:24.991513 (+ 257us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:24.991517 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:24.991981 (+ 464us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:24.992169 (+ 188us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.992175 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.993055 (+ 880us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.993062 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:24.993456 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:24.993462 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.993592 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.994167 (+ 575us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:24.994181 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:24.995666 (+ 1485us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:24.997413 (+ 1747us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.997422 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.997427 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.997640 (+ 213us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.997927 (+ 287us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:24.997931 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:24.997935 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:24.997984 (+ 49us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:24.998332 (+ 348us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:24.998340 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:24.998436 (+ 96us) client_negotiation.cc:770] Sending connection context
0430 08:01:24.998550 (+ 114us) client_negotiation.cc:241] Negotiation successful
0430 08:01:24.998713 (+ 163us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":121,"threads_started":1}
I20260430 08:01:24.998965 22427 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:24.990707 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.13.94.190:45879)
0430 08:01:24.991073 (+ 366us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:24.991078 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:24.991183 (+ 105us) server_negotiation.cc:408] Connection header received
0430 08:01:24.991355 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:24.991359 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:24.991403 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:24.991496 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:24.992314 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.992911 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:24.993567 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:24.993739 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:24.995789 (+ 2050us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:24.995816 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:24.995818 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:24.995838 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:24.997250 (+ 1412us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.997738 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.997743 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.997765 (+ 22us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.997816 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:24.998076 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:24.998079 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:24.998081 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:24.998219 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:24.998318 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:24.998594 (+ 276us) server_negotiation.cc:300] Negotiation successful
0430 08:01:24.998781 (+ 187us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
I20260430 08:01:25.003505 21871 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:39714:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260430 08:01:25.004269 21871 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:01:25.009744 22010 tablet_service.cc:1511] Processing CreateTablet for tablet 8598914feb01430397467ac645bc2917 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82f8c93d2d9447059211f6cbfdb5f954]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 08:01:25.009948 22152 tablet_service.cc:1511] Processing CreateTablet for tablet 8598914feb01430397467ac645bc2917 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82f8c93d2d9447059211f6cbfdb5f954]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 08:01:25.010219 22010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8598914feb01430397467ac645bc2917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:25.010377 22152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8598914feb01430397467ac645bc2917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:25.012769 22428 tablet_bootstrap.cc:492] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: Bootstrap starting.
I20260430 08:01:25.013342 22429 tablet_bootstrap.cc:492] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: Bootstrap starting.
I20260430 08:01:25.013515 22294 tablet_service.cc:1511] Processing CreateTablet for tablet 8598914feb01430397467ac645bc2917 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82f8c93d2d9447059211f6cbfdb5f954]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260430 08:01:25.013799 22294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8598914feb01430397467ac645bc2917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:25.013799 22428 tablet_bootstrap.cc:654] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:25.014364 22429 tablet_bootstrap.cc:654] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:25.015085 22428 tablet_bootstrap.cc:492] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: No bootstrap required, opened a new log
I20260430 08:01:25.015154 22428 ts_tablet_manager.cc:1403] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:25.015542 22428 raft_consensus.cc:359] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.015641 22428 raft_consensus.cc:385] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:25.015664 22428 raft_consensus.cc:740] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7792d82ea35a4105a892c66260f1b546, State: Initialized, Role: FOLLOWER
I20260430 08:01:25.015794 22428 consensus_queue.cc:260] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.016106 22428 ts_tablet_manager.cc:1434] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:01:25.016146 22432 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.016256 22432 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:01:25.016935 22433 tablet_bootstrap.cc:492] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: Bootstrap starting.
I20260430 08:01:25.016963 22429 tablet_bootstrap.cc:492] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: No bootstrap required, opened a new log
I20260430 08:01:25.017059 22429 ts_tablet_manager.cc:1403] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260430 08:01:25.017558 22429 raft_consensus.cc:359] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.017669 22429 raft_consensus.cc:385] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:25.017694 22429 raft_consensus.cc:740] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfb7409eeab4884a492e47bb318e059, State: Initialized, Role: FOLLOWER
I20260430 08:01:25.017828 22429 consensus_queue.cc:260] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.018092 22433 tablet_bootstrap.cc:654] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:25.018177 22429 ts_tablet_manager.cc:1434] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260430 08:01:25.018664 22434 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.018790 22434 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:01:25.019876 22433 tablet_bootstrap.cc:492] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: No bootstrap required, opened a new log
I20260430 08:01:25.020005 22433 ts_tablet_manager.cc:1403] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260430 08:01:25.020609 22433 raft_consensus.cc:359] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.020771 22433 raft_consensus.cc:385] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:25.020823 22433 raft_consensus.cc:740] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f661fe8f5e241acbf8b5482b32be069, State: Initialized, Role: FOLLOWER
I20260430 08:01:25.020968 22433 consensus_queue.cc:260] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.021299 22433 ts_tablet_manager.cc:1434] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:01:25.021258 22395 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.021425 22395 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:01:25.063292 22434 raft_consensus.cc:493] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:01:25.063438 22434 raft_consensus.cc:515] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.064518 22434 leader_election.cc:290] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179), 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691)
I20260430 08:01:25.068054 22436 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.064914 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:36179 (local address 127.13.94.129:53095)
0430 08:01:25.065355 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:25.065371 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:25.065501 (+ 130us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:25.065854 (+ 353us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:25.065856 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:25.065873 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:25.066139 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.066144 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.067030 (+ 886us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.067034 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:25.067692 (+ 658us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.067699 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.067789 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.067803 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 08:01:25.067839 (+ 36us) client_negotiation.cc:241] Negotiation successful
0430 08:01:25.067880 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":118,"threads_started":1}
I20260430 08:01:25.068519 22437 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.065356 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:42691 (local address 127.13.94.129:43829)
0430 08:01:25.065625 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:25.065639 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:25.065750 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:25.066067 (+ 317us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:25.066070 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:25.066092 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:25.066402 (+ 310us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.066411 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.067251 (+ 840us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.067254 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:25.067897 (+ 643us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.067905 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.068050 (+ 145us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.068064 (+ 14us) client_negotiation.cc:770] Sending connection context
0430 08:01:25.068328 (+ 264us) client_negotiation.cc:241] Negotiation successful
0430 08:01:25.068387 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":110,"threads_started":1}
I20260430 08:01:25.068563 22387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.065021 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:53095 (local address 127.13.94.130:36179)
0430 08:01:25.065341 (+ 320us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.065346 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.065452 (+ 106us) server_negotiation.cc:408] Connection header received
0430 08:01:25.065654 (+ 202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.065658 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.065701 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.065744 (+ 43us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:25.066309 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.066875 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.067821 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.068311 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.068350 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.068402 (+ 52us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.068457 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260430 08:01:25.068778 22438 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.065474 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:43829 (local address 127.13.94.131:42691)
0430 08:01:25.065728 (+ 254us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.065732 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.065746 (+ 14us) server_negotiation.cc:408] Connection header received
0430 08:01:25.065901 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.065905 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.065955 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.066054 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:25.066537 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.067118 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.068038 (+ 920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.068456 (+ 418us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.068484 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.068528 (+ 44us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.068562 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260430 08:01:25.069169 22308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8598914feb01430397467ac645bc2917" candidate_uuid: "4bfb7409eeab4884a492e47bb318e059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f661fe8f5e241acbf8b5482b32be069" is_pre_election: true
I20260430 08:01:25.069164 22172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8598914feb01430397467ac645bc2917" candidate_uuid: "4bfb7409eeab4884a492e47bb318e059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792d82ea35a4105a892c66260f1b546" is_pre_election: true
I20260430 08:01:25.069324 22172 raft_consensus.cc:2468] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bfb7409eeab4884a492e47bb318e059 in term 0.
I20260430 08:01:25.069326 22308 raft_consensus.cc:2468] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bfb7409eeab4884a492e47bb318e059 in term 0.
I20260430 08:01:25.069691 21961 leader_election.cc:304] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfb7409eeab4884a492e47bb318e059, 6f661fe8f5e241acbf8b5482b32be069; no voters:
I20260430 08:01:25.069952 22434 raft_consensus.cc:2804] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:01:25.070008 22434 raft_consensus.cc:493] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:01:25.070036 22434 raft_consensus.cc:3060] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:25.070995 22434 raft_consensus.cc:515] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.071388 22434 leader_election.cc:290] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [CANDIDATE]: Term 1 election: Requested vote from peers 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179), 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691)
I20260430 08:01:25.071810 22172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8598914feb01430397467ac645bc2917" candidate_uuid: "4bfb7409eeab4884a492e47bb318e059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792d82ea35a4105a892c66260f1b546"
I20260430 08:01:25.071882 22308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8598914feb01430397467ac645bc2917" candidate_uuid: "4bfb7409eeab4884a492e47bb318e059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f661fe8f5e241acbf8b5482b32be069"
I20260430 08:01:25.071949 22172 raft_consensus.cc:3060] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:25.072036 22308 raft_consensus.cc:3060] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:25.073078 22308 raft_consensus.cc:2468] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bfb7409eeab4884a492e47bb318e059 in term 1.
I20260430 08:01:25.073174 22172 raft_consensus.cc:2468] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bfb7409eeab4884a492e47bb318e059 in term 1.
I20260430 08:01:25.073545 21961 leader_election.cc:304] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfb7409eeab4884a492e47bb318e059, 7792d82ea35a4105a892c66260f1b546; no voters:
I20260430 08:01:25.073776 22434 raft_consensus.cc:2804] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:25.074000 22434 raft_consensus.cc:697] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [term 1 LEADER]: Becoming Leader. State: Replica: 4bfb7409eeab4884a492e47bb318e059, State: Running, Role: LEADER
I20260430 08:01:25.074263 22434 consensus_queue.cc:237] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.075542 22439 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: TxnStatusTablet state changed. Reason: New leader 4bfb7409eeab4884a492e47bb318e059. Latest consensus state: current_term: 1 leader_uuid: "4bfb7409eeab4884a492e47bb318e059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.075657 22439 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: This TxnStatusTablet replica's current role is: LEADER
I20260430 08:01:25.076117 22441 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260430 08:01:25.076274 22441 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260430 08:01:25.076846 21871 catalog_manager.cc:5671] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129). New cstate: current_term: 1 leader_uuid: "4bfb7409eeab4884a492e47bb318e059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } health_report { overall_health: UNKNOWN } } }
I20260430 08:01:25.099040 22426 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.095098 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:45127 (local address 127.0.0.1:34856)
0430 08:01:25.095281 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:25.095297 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:25.095426 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:25.095791 (+ 365us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:25.095794 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:25.095802 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:01:25.096014 (+ 212us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.096019 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.097161 (+ 1142us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.097165 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:25.097633 (+ 468us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.097641 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.097746 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.097785 (+ 39us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.098691 (+ 906us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.098704 (+ 13us) client_negotiation.cc:770] Sending connection context
0430 08:01:25.098777 (+ 73us) client_negotiation.cc:241] Negotiation successful
0430 08:01:25.098846 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260430 08:01:25.099030 22386 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.095278 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.13.94.129:45127)
0430 08:01:25.095460 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.095465 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.095481 (+ 16us) server_negotiation.cc:408] Connection header received
0430 08:01:25.095564 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.095567 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.095638 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.095746 (+ 108us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:25.096130 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.096914 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.097754 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.097951 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.098042 (+ 91us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.098561 (+ 519us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.098671 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.098811 (+ 140us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.098862 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260430 08:01:25.102254 22308 raft_consensus.cc:1275] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Refusing update from remote peer 4bfb7409eeab4884a492e47bb318e059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:25.102501 22172 raft_consensus.cc:1275] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Refusing update from remote peer 4bfb7409eeab4884a492e47bb318e059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:25.102734 22434 consensus_queue.cc:1048] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:25.103118 22439 consensus_queue.cc:1048] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:25.104367 22395 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: TxnStatusTablet state changed. Reason: New leader 4bfb7409eeab4884a492e47bb318e059. Latest consensus state: current_term: 1 leader_uuid: "4bfb7409eeab4884a492e47bb318e059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.104477 22395 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:01:25.106863 22432 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: TxnStatusTablet state changed. Reason: New leader 4bfb7409eeab4884a492e47bb318e059. Latest consensus state: current_term: 1 leader_uuid: "4bfb7409eeab4884a492e47bb318e059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.106989 22432 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:01:25.107122 22434 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bfb7409eeab4884a492e47bb318e059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.107229 22434 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: This TxnStatusTablet replica's current role is: LEADER
I20260430 08:01:25.107679 22395 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bfb7409eeab4884a492e47bb318e059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.107781 22395 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 6f661fe8f5e241acbf8b5482b32be069: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:01:25.109670 22445 mvcc.cc:204] Tried to move back new op lower bound from 7280787804573696000 to 7280787804466675712. Current Snapshot: MvccSnapshot[applied={T|T < 7280787804573696000}]
I20260430 08:01:25.113655 22439 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bfb7409eeab4884a492e47bb318e059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.113787 22439 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 4bfb7409eeab4884a492e47bb318e059: This TxnStatusTablet replica's current role is: LEADER
I20260430 08:01:25.114511 22447 mvcc.cc:204] Tried to move back new op lower bound from 7280787804573696000 to 7280787804466675712. Current Snapshot: MvccSnapshot[applied={T|T < 7280787804573696000}]
I20260430 08:01:25.114866 22432 tablet_replica.cc:442] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bfb7409eeab4884a492e47bb318e059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } } }
I20260430 08:01:25.114977 22432 tablet_replica.cc:445] T 8598914feb01430397467ac645bc2917 P 7792d82ea35a4105a892c66260f1b546: This TxnStatusTablet replica's current role is: FOLLOWER
I20260430 08:01:25.117985 22438 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.112859 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.13.94.131:42691)
0430 08:01:25.113156 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.113160 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.113178 (+ 18us) server_negotiation.cc:408] Connection header received
0430 08:01:25.113272 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.113276 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.113337 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.113448 (+ 111us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:25.115433 (+ 1985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.116215 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.116955 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.117169 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.117223 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.117599 (+ 376us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.117720 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.117808 (+ 88us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.117852 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260430 08:01:25.128556 22386 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.125390 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:51101 (local address 127.13.94.129:45127)
0430 08:01:25.125519 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.125524 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.125779 (+ 255us) server_negotiation.cc:408] Connection header received
0430 08:01:25.125910 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.125913 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.125964 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.126036 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:25.126511 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.126964 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.127883 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.128076 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.128118 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.128191 (+ 73us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.128258 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.128399 (+ 141us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.128448 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260430 08:01:25.128603 22456 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.125225 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:45127 (local address 127.13.94.131:51101)
0430 08:01:25.125682 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:25.125697 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:25.125809 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:25.126074 (+ 265us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:25.126077 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:25.126086 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:01:25.126396 (+ 310us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.126403 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.127080 (+ 677us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.127083 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:25.127745 (+ 662us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.127754 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.127890 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.127934 (+ 44us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.128294 (+ 360us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.128299 (+ 5us) client_negotiation.cc:770] Sending connection context
0430 08:01:25.128370 (+ 71us) client_negotiation.cc:241] Negotiation successful
0430 08:01:25.128438 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":110,"threads_started":1}
I20260430 08:01:25.137125 22030 raft_consensus.cc:1275] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Refusing update from remote peer 6f661fe8f5e241acbf8b5482b32be069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:25.137539 22172 raft_consensus.cc:1275] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Refusing update from remote peer 6f661fe8f5e241acbf8b5482b32be069: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:25.137794 22419 consensus_queue.cc:1048] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:25.137970 22395 consensus_queue.cc:1048] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:25.160166 22462 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.156537 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:42691 (local address 127.13.94.129:40259)
0430 08:01:25.156889 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:25.156902 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:25.156995 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:25.157330 (+ 335us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:25.157334 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:25.157343 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:01:25.157652 (+ 309us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.157658 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.158656 (+ 998us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.158659 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:25.159344 (+ 685us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.159353 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.159467 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.159510 (+ 43us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.159862 (+ 352us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.159868 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 08:01:25.159937 (+ 69us) client_negotiation.cc:241] Negotiation successful
0430 08:01:25.160003 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
I20260430 08:01:25.160115 22438 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.156636 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:40259 (local address 127.13.94.131:42691)
0430 08:01:25.156808 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.156812 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.156974 (+ 162us) server_negotiation.cc:408] Connection header received
0430 08:01:25.157085 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.157088 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.157141 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.157225 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:25.157776 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.158521 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.159481 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.159632 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.159672 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.159753 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.159827 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.159971 (+ 144us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.160007 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260430 08:01:25.189338 21871 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:25.189543 21871 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:25.192332 21871 catalog_manager.cc:5958] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1: Sending DeleteTablet for 3 replicas of tablet 6edf7a6ebd34478891ea4c5cf6d37239
I20260430 08:01:25.193306 22152 tablet_service.cc:1558] Processing DeleteTablet for tablet 6edf7a6ebd34478891ea4c5cf6d37239 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:25 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:37040
I20260430 08:01:25.193758 22465 tablet_replica.cc:333] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: stopping tablet replica
I20260430 08:01:25.193840 22010 tablet_service.cc:1558] Processing DeleteTablet for tablet 6edf7a6ebd34478891ea4c5cf6d37239 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:25 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:34850
I20260430 08:01:25.194093 22465 raft_consensus.cc:2243] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:25.194311 22465 raft_consensus.cc:2272] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:25.194334 22294 tablet_service.cc:1558] Processing DeleteTablet for tablet 6edf7a6ebd34478891ea4c5cf6d37239 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:25 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42634
I20260430 08:01:25.195307 22465 ts_tablet_manager.cc:1916] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:25.198622 22465 ts_tablet_manager.cc:1929] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:01:25.198848 22465 log.cc:1199] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/6edf7a6ebd34478891ea4c5cf6d37239
I20260430 08:01:25.199342 22465 ts_tablet_manager.cc:1950] T 6edf7a6ebd34478891ea4c5cf6d37239 P 7792d82ea35a4105a892c66260f1b546: Deleting consensus metadata
I20260430 08:01:25.200384 21855 catalog_manager.cc:5002] TS 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179): tablet 6edf7a6ebd34478891ea4c5cf6d37239 (table test-table [id=c86764180a094c9b8a423fd8c1edb52f]) successfully deleted
I20260430 08:01:25.200870 22467 tablet_replica.cc:333] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: stopping tablet replica
I20260430 08:01:25.201089 22467 raft_consensus.cc:2243] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:01:25.201527 22467 raft_consensus.cc:2272] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:25.202405 22466 tablet_replica.cc:333] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: stopping tablet replica
I20260430 08:01:25.202535 22467 ts_tablet_manager.cc:1916] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:25.202543 22466 raft_consensus.cc:2243] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:25.202675 22466 raft_consensus.cc:2272] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:25.204934 22466 ts_tablet_manager.cc:1916] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:25.206179 22467 ts_tablet_manager.cc:1929] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:01:25.206323 22467 log.cc:1199] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/6edf7a6ebd34478891ea4c5cf6d37239
I20260430 08:01:25.206682 22467 ts_tablet_manager.cc:1950] T 6edf7a6ebd34478891ea4c5cf6d37239 P 6f661fe8f5e241acbf8b5482b32be069: Deleting consensus metadata
I20260430 08:01:25.207595 21855 catalog_manager.cc:5002] TS 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691): tablet 6edf7a6ebd34478891ea4c5cf6d37239 (table test-table [id=c86764180a094c9b8a423fd8c1edb52f]) successfully deleted
I20260430 08:01:25.209457 22466 ts_tablet_manager.cc:1929] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:01:25.209539 22466 log.cc:1199] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/6edf7a6ebd34478891ea4c5cf6d37239
I20260430 08:01:25.209815 22466 ts_tablet_manager.cc:1950] T 6edf7a6ebd34478891ea4c5cf6d37239 P 4bfb7409eeab4884a492e47bb318e059: Deleting consensus metadata
I20260430 08:01:25.210620 21857 catalog_manager.cc:5002] TS 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127): tablet 6edf7a6ebd34478891ea4c5cf6d37239 (table test-table [id=c86764180a094c9b8a423fd8c1edb52f]) successfully deleted
Apr 30 08:01:25 dist-test-slave-9bxp krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 30 08:01:25 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:25 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
W20260430 08:01:25.233865 22427 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:25.226283 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.13.94.190:45879)
0430 08:01:25.226448 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.226455 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.226618 (+ 163us) server_negotiation.cc:408] Connection header received
0430 08:01:25.226767 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.226772 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.226833 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.226934 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:25.228050 (+ 1116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.228892 (+ 842us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.229843 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.230051 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.233718 (+ 3667us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39720: BlockingRecv error: recv got EOF from 127.0.0.1:39720 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Apr 30 08:01:25 dist-test-slave-9bxp krb5kdc[21823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM
I20260430 08:01:25.246325 22427 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.235838 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.13.94.190:45879)
0430 08:01:25.235973 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.235978 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.236463 (+ 485us) server_negotiation.cc:408] Connection header received
0430 08:01:25.236587 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.236592 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.236651 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.236787 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:25.237843 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.238685 (+ 842us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.239564 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.240111 (+ 547us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.242567 (+ 2456us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:25.242591 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:25.242595 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:25.242634 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:25.244524 (+ 1890us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:25.245024 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:25.245029 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:25.245031 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:25.245090 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:25.245370 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:25.245374 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:25.245377 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:25.245572 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:25.245725 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.245972 (+ 247us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.246112 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 08:01:25.249688 21871 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:01:25.250239 21871 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:01:25.257258 22152 tablet_service.cc:1511] Processing CreateTablet for tablet 1693342168ad433d8d39343a5867fcbf (DEFAULT_TABLE table=test-table [id=bbf9cf882529468da75c199a46c90e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:25.257611 22152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1693342168ad433d8d39343a5867fcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:25.260422 22294 tablet_service.cc:1511] Processing CreateTablet for tablet 1693342168ad433d8d39343a5867fcbf (DEFAULT_TABLE table=test-table [id=bbf9cf882529468da75c199a46c90e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:25.261227 22294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1693342168ad433d8d39343a5867fcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:25.263520 22010 tablet_service.cc:1511] Processing CreateTablet for tablet 1693342168ad433d8d39343a5867fcbf (DEFAULT_TABLE table=test-table [id=bbf9cf882529468da75c199a46c90e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:25.263937 22010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1693342168ad433d8d39343a5867fcbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:25.276564 22429 tablet_bootstrap.cc:492] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: Bootstrap starting.
I20260430 08:01:25.277661 22433 tablet_bootstrap.cc:492] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: Bootstrap starting.
I20260430 08:01:25.277750 22429 tablet_bootstrap.cc:654] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:25.278834 22433 tablet_bootstrap.cc:654] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:25.281024 22428 tablet_bootstrap.cc:492] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: Bootstrap starting.
I20260430 08:01:25.282460 22428 tablet_bootstrap.cc:654] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:25.289728 22428 tablet_bootstrap.cc:492] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: No bootstrap required, opened a new log
I20260430 08:01:25.289875 22428 ts_tablet_manager.cc:1403] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260430 08:01:25.290480 22428 raft_consensus.cc:359] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.290618 22428 raft_consensus.cc:385] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:25.290659 22428 raft_consensus.cc:740] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7792d82ea35a4105a892c66260f1b546, State: Initialized, Role: FOLLOWER
I20260430 08:01:25.290865 22428 consensus_queue.cc:260] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.293501 22428 ts_tablet_manager.cc:1434] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260430 08:01:25.293612 22433 tablet_bootstrap.cc:492] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: No bootstrap required, opened a new log
I20260430 08:01:25.293715 22433 ts_tablet_manager.cc:1403] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260430 08:01:25.294235 22433 raft_consensus.cc:359] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.294355 22433 raft_consensus.cc:385] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:25.294391 22433 raft_consensus.cc:740] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f661fe8f5e241acbf8b5482b32be069, State: Initialized, Role: FOLLOWER
I20260430 08:01:25.295428 22433 consensus_queue.cc:260] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.297523 22429 tablet_bootstrap.cc:492] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: No bootstrap required, opened a new log
I20260430 08:01:25.297626 22429 ts_tablet_manager.cc:1403] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20260430 08:01:25.297777 22433 ts_tablet_manager.cc:1434] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260430 08:01:25.298170 22429 raft_consensus.cc:359] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.298324 22429 raft_consensus.cc:385] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:25.298367 22429 raft_consensus.cc:740] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfb7409eeab4884a492e47bb318e059, State: Initialized, Role: FOLLOWER
I20260430 08:01:25.298518 22429 consensus_queue.cc:260] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.298801 22429 ts_tablet_manager.cc:1434] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260430 08:01:25.429419 22434 raft_consensus.cc:493] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:01:25.429558 22434 raft_consensus.cc:515] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.430054 22434 leader_election.cc:290] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179), 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691)
I20260430 08:01:25.430661 22308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1693342168ad433d8d39343a5867fcbf" candidate_uuid: "4bfb7409eeab4884a492e47bb318e059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f661fe8f5e241acbf8b5482b32be069" is_pre_election: true
I20260430 08:01:25.430675 22172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1693342168ad433d8d39343a5867fcbf" candidate_uuid: "4bfb7409eeab4884a492e47bb318e059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792d82ea35a4105a892c66260f1b546" is_pre_election: true
I20260430 08:01:25.430835 22308 raft_consensus.cc:2468] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bfb7409eeab4884a492e47bb318e059 in term 0.
I20260430 08:01:25.430840 22172 raft_consensus.cc:2468] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bfb7409eeab4884a492e47bb318e059 in term 0.
I20260430 08:01:25.431183 21961 leader_election.cc:304] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfb7409eeab4884a492e47bb318e059, 7792d82ea35a4105a892c66260f1b546; no voters:
I20260430 08:01:25.431398 22434 raft_consensus.cc:2804] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:01:25.431463 22434 raft_consensus.cc:493] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:01:25.431505 22434 raft_consensus.cc:3060] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:25.432457 22434 raft_consensus.cc:515] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.432946 22434 leader_election.cc:290] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [CANDIDATE]: Term 1 election: Requested vote from peers 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179), 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691)
I20260430 08:01:25.433426 22172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1693342168ad433d8d39343a5867fcbf" candidate_uuid: "4bfb7409eeab4884a492e47bb318e059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7792d82ea35a4105a892c66260f1b546"
I20260430 08:01:25.433434 22308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1693342168ad433d8d39343a5867fcbf" candidate_uuid: "4bfb7409eeab4884a492e47bb318e059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f661fe8f5e241acbf8b5482b32be069"
I20260430 08:01:25.433562 22172 raft_consensus.cc:3060] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:25.433562 22308 raft_consensus.cc:3060] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:25.434696 22308 raft_consensus.cc:2468] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bfb7409eeab4884a492e47bb318e059 in term 1.
I20260430 08:01:25.434695 22172 raft_consensus.cc:2468] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bfb7409eeab4884a492e47bb318e059 in term 1.
I20260430 08:01:25.435104 21961 leader_election.cc:304] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfb7409eeab4884a492e47bb318e059, 6f661fe8f5e241acbf8b5482b32be069; no voters:
I20260430 08:01:25.435374 22434 raft_consensus.cc:2804] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:25.435492 22434 raft_consensus.cc:697] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 1 LEADER]: Becoming Leader. State: Replica: 4bfb7409eeab4884a492e47bb318e059, State: Running, Role: LEADER
I20260430 08:01:25.435659 22434 consensus_queue.cc:237] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.437748 21871 catalog_manager.cc:5671] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129). New cstate: current_term: 1 leader_uuid: "4bfb7409eeab4884a492e47bb318e059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } health_report { overall_health: UNKNOWN } } }
I20260430 08:01:25.454126 22386 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.451449 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.13.94.129:45127)
0430 08:01:25.451583 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.451589 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.451606 (+ 17us) server_negotiation.cc:408] Connection header received
0430 08:01:25.451664 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.451672 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.451719 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.451803 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:25.452171 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.452648 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.453361 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.453556 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.453593 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.453661 (+ 68us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.453747 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.453897 (+ 150us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.453940 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260430 08:01:25.456622 22308 raft_consensus.cc:1275] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Refusing update from remote peer 4bfb7409eeab4884a492e47bb318e059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:25.456686 22172 raft_consensus.cc:1275] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Refusing update from remote peer 4bfb7409eeab4884a492e47bb318e059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:25.457252 22457 consensus_queue.cc:1048] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:25.457414 22434 consensus_queue.cc:1048] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:25.467332 21871 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:25.467736 21871 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:25.470037 21871 catalog_manager.cc:5958] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1: Sending DeleteTablet for 3 replicas of tablet 1693342168ad433d8d39343a5867fcbf
I20260430 08:01:25.470646 22010 tablet_service.cc:1558] Processing DeleteTablet for tablet 1693342168ad433d8d39343a5867fcbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:25 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:34850
I20260430 08:01:25.470777 22152 tablet_service.cc:1558] Processing DeleteTablet for tablet 1693342168ad433d8d39343a5867fcbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:25 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:37040
I20260430 08:01:25.470840 22466 tablet_replica.cc:333] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: stopping tablet replica
I20260430 08:01:25.470906 22294 tablet_service.cc:1558] Processing DeleteTablet for tablet 1693342168ad433d8d39343a5867fcbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:25 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42634
I20260430 08:01:25.470970 22466 raft_consensus.cc:2243] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:01:25.471258 22466 raft_consensus.cc:2272] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:25.471472 22465 tablet_replica.cc:333] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: stopping tablet replica
I20260430 08:01:25.471589 22465 raft_consensus.cc:2243] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:25.471712 22465 raft_consensus.cc:2272] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:25.471925 22467 tablet_replica.cc:333] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: stopping tablet replica
I20260430 08:01:25.472070 22467 raft_consensus.cc:2243] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:25.472069 22466 ts_tablet_manager.cc:1916] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:25.472203 22467 raft_consensus.cc:2272] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:25.472569 22465 ts_tablet_manager.cc:1916] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:25.474470 22466 ts_tablet_manager.cc:1929] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:25.474539 22466 log.cc:1199] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/1693342168ad433d8d39343a5867fcbf
I20260430 08:01:25.474778 22465 ts_tablet_manager.cc:1929] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:25.474799 22466 ts_tablet_manager.cc:1950] T 1693342168ad433d8d39343a5867fcbf P 4bfb7409eeab4884a492e47bb318e059: Deleting consensus metadata
I20260430 08:01:25.474851 22465 log.cc:1199] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/1693342168ad433d8d39343a5867fcbf
I20260430 08:01:25.475101 22465 ts_tablet_manager.cc:1950] T 1693342168ad433d8d39343a5867fcbf P 7792d82ea35a4105a892c66260f1b546: Deleting consensus metadata
I20260430 08:01:25.475536 21857 catalog_manager.cc:5002] TS 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127): tablet 1693342168ad433d8d39343a5867fcbf (table test-table [id=bbf9cf882529468da75c199a46c90e1f]) successfully deleted
I20260430 08:01:25.476248 21855 catalog_manager.cc:5002] TS 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179): tablet 1693342168ad433d8d39343a5867fcbf (table test-table [id=bbf9cf882529468da75c199a46c90e1f]) successfully deleted
I20260430 08:01:25.477201 21871 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260430 08:01:25.477586 22467 ts_tablet_manager.cc:1916] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260430 08:01:25.477681 21871 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:01:25.479915 22467 ts_tablet_manager.cc:1929] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:25.480055 22467 log.cc:1199] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/1693342168ad433d8d39343a5867fcbf
I20260430 08:01:25.480337 22467 ts_tablet_manager.cc:1950] T 1693342168ad433d8d39343a5867fcbf P 6f661fe8f5e241acbf8b5482b32be069: Deleting consensus metadata
I20260430 08:01:25.483081 22010 tablet_service.cc:1511] Processing CreateTablet for tablet bc0aa0ca63dd4b75a08970b0c9b1827f (DEFAULT_TABLE table=test-table [id=10b268ddea314442a277374a58ac3dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:25.483397 22010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0aa0ca63dd4b75a08970b0c9b1827f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:25.483423 22294 tablet_service.cc:1511] Processing CreateTablet for tablet bc0aa0ca63dd4b75a08970b0c9b1827f (DEFAULT_TABLE table=test-table [id=10b268ddea314442a277374a58ac3dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:25.483448 22152 tablet_service.cc:1511] Processing CreateTablet for tablet bc0aa0ca63dd4b75a08970b0c9b1827f (DEFAULT_TABLE table=test-table [id=10b268ddea314442a277374a58ac3dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:25.483651 22294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0aa0ca63dd4b75a08970b0c9b1827f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:25.483701 22152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0aa0ca63dd4b75a08970b0c9b1827f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:25.485741 22429 tablet_bootstrap.cc:492] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: Bootstrap starting.
I20260430 08:01:25.485862 22428 tablet_bootstrap.cc:492] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: Bootstrap starting.
I20260430 08:01:25.485873 22433 tablet_bootstrap.cc:492] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: Bootstrap starting.
I20260430 08:01:25.486706 22428 tablet_bootstrap.cc:654] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:25.486714 22429 tablet_bootstrap.cc:654] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:25.486714 22433 tablet_bootstrap.cc:654] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:25.487937 21855 catalog_manager.cc:5002] TS 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691): tablet 1693342168ad433d8d39343a5867fcbf (table test-table [id=bbf9cf882529468da75c199a46c90e1f]) successfully deleted
I20260430 08:01:25.488045 22429 tablet_bootstrap.cc:492] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: No bootstrap required, opened a new log
I20260430 08:01:25.488121 22429 ts_tablet_manager.cc:1403] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260430 08:01:25.488543 22429 raft_consensus.cc:359] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.488646 22429 raft_consensus.cc:385] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:25.488678 22429 raft_consensus.cc:740] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfb7409eeab4884a492e47bb318e059, State: Initialized, Role: FOLLOWER
I20260430 08:01:25.488833 22429 consensus_queue.cc:260] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.489112 22429 ts_tablet_manager.cc:1434] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:01:25.490073 22433 tablet_bootstrap.cc:492] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: No bootstrap required, opened a new log
I20260430 08:01:25.490154 22433 ts_tablet_manager.cc:1403] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260430 08:01:25.490569 22433 raft_consensus.cc:359] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.490664 22433 raft_consensus.cc:385] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:25.490695 22433 raft_consensus.cc:740] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f661fe8f5e241acbf8b5482b32be069, State: Initialized, Role: FOLLOWER
I20260430 08:01:25.490819 22433 consensus_queue.cc:260] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.491070 22433 ts_tablet_manager.cc:1434] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260430 08:01:25.500979 22428 tablet_bootstrap.cc:492] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: No bootstrap required, opened a new log
I20260430 08:01:25.501108 22428 ts_tablet_manager.cc:1403] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260430 08:01:25.501603 22428 raft_consensus.cc:359] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.501761 22428 raft_consensus.cc:385] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:25.501798 22428 raft_consensus.cc:740] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7792d82ea35a4105a892c66260f1b546, State: Initialized, Role: FOLLOWER
I20260430 08:01:25.501940 22428 consensus_queue.cc:260] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.502228 22428 ts_tablet_manager.cc:1434] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260430 08:01:25.536763 22432 raft_consensus.cc:493] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:01:25.536986 22432 raft_consensus.cc:515] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.537575 22432 leader_election.cc:290] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127), 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691)
I20260430 08:01:25.538064 22308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0aa0ca63dd4b75a08970b0c9b1827f" candidate_uuid: "7792d82ea35a4105a892c66260f1b546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f661fe8f5e241acbf8b5482b32be069" is_pre_election: true
I20260430 08:01:25.538059 22030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0aa0ca63dd4b75a08970b0c9b1827f" candidate_uuid: "7792d82ea35a4105a892c66260f1b546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfb7409eeab4884a492e47bb318e059" is_pre_election: true
I20260430 08:01:25.538230 22030 raft_consensus.cc:2468] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7792d82ea35a4105a892c66260f1b546 in term 0.
I20260430 08:01:25.538244 22308 raft_consensus.cc:2468] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7792d82ea35a4105a892c66260f1b546 in term 0.
I20260430 08:01:25.538582 22105 leader_election.cc:304] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bfb7409eeab4884a492e47bb318e059, 7792d82ea35a4105a892c66260f1b546; no voters:
I20260430 08:01:25.538810 22432 raft_consensus.cc:2804] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:01:25.538892 22432 raft_consensus.cc:493] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:01:25.538931 22432 raft_consensus.cc:3060] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:25.540146 22432 raft_consensus.cc:515] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.540552 22432 leader_election.cc:290] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [CANDIDATE]: Term 1 election: Requested vote from peers 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127), 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691)
I20260430 08:01:25.540958 22308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0aa0ca63dd4b75a08970b0c9b1827f" candidate_uuid: "7792d82ea35a4105a892c66260f1b546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f661fe8f5e241acbf8b5482b32be069"
I20260430 08:01:25.541087 22308 raft_consensus.cc:3060] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:25.541088 22030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0aa0ca63dd4b75a08970b0c9b1827f" candidate_uuid: "7792d82ea35a4105a892c66260f1b546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfb7409eeab4884a492e47bb318e059"
I20260430 08:01:25.541249 22030 raft_consensus.cc:3060] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:25.541965 22308 raft_consensus.cc:2468] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7792d82ea35a4105a892c66260f1b546 in term 1.
I20260430 08:01:25.542271 22030 raft_consensus.cc:2468] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7792d82ea35a4105a892c66260f1b546 in term 1.
I20260430 08:01:25.542286 22103 leader_election.cc:304] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f661fe8f5e241acbf8b5482b32be069, 7792d82ea35a4105a892c66260f1b546; no voters:
I20260430 08:01:25.542586 22432 raft_consensus.cc:2804] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:25.542762 22432 raft_consensus.cc:697] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 1 LEADER]: Becoming Leader. State: Replica: 7792d82ea35a4105a892c66260f1b546, State: Running, Role: LEADER
I20260430 08:01:25.542905 22432 consensus_queue.cc:237] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } }
I20260430 08:01:25.544584 21870 catalog_manager.cc:5671] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7792d82ea35a4105a892c66260f1b546 (127.13.94.130). New cstate: current_term: 1 leader_uuid: "7792d82ea35a4105a892c66260f1b546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7792d82ea35a4105a892c66260f1b546" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 36179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 } health_report { overall_health: UNKNOWN } } }
I20260430 08:01:25.568780 22387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.565716 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.13.94.130:36179)
0430 08:01:25.565845 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.565850 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.565867 (+ 17us) server_negotiation.cc:408] Connection header received
0430 08:01:25.565998 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.566002 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.566059 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.566146 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:25.566553 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.567312 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.567979 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.568178 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.568231 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.568314 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.568392 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.568572 (+ 180us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.568618 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260430 08:01:25.579217 22495 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.575945 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:45127 (local address 127.13.94.130:45473)
0430 08:01:25.576237 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:25.576253 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:25.576410 (+ 157us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:25.576668 (+ 258us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:25.576671 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:25.576680 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:01:25.576995 (+ 315us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.577004 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.578015 (+ 1011us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.578019 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:25.578457 (+ 438us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.578464 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.578558 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.578595 (+ 37us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.578988 (+ 393us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.578991 (+ 3us) client_negotiation.cc:770] Sending connection context
0430 08:01:25.579043 (+ 52us) client_negotiation.cc:241] Negotiation successful
0430 08:01:25.579098 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
I20260430 08:01:25.579272 22386 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.576098 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:45473 (local address 127.13.94.129:45127)
0430 08:01:25.576232 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.576236 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.576348 (+ 112us) server_negotiation.cc:408] Connection header received
0430 08:01:25.576513 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.576517 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.576573 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.576657 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:25.577133 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.577890 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.578575 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.578769 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.578813 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.578900 (+ 87us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.578979 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.579086 (+ 107us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.579130 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260430 08:01:25.585680 22030 raft_consensus.cc:1275] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Refusing update from remote peer 7792d82ea35a4105a892c66260f1b546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:25.585709 22308 raft_consensus.cc:1275] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Refusing update from remote peer 7792d82ea35a4105a892c66260f1b546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:25.586069 22491 consensus_queue.cc:1048] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f661fe8f5e241acbf8b5482b32be069" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 42691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:25.586229 22432 consensus_queue.cc:1048] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bfb7409eeab4884a492e47bb318e059" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 45127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:25.606058 22462 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.602312 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:36179 (local address 127.13.94.129:53489)
0430 08:01:25.602485 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:25.602503 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:25.602616 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:25.603160 (+ 544us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:25.603163 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:25.603174 (+ 11us) client_negotiation.cc:190] Negotiated authn=TOKEN
0430 08:01:25.603393 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.603400 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.604136 (+ 736us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.604141 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:25.604764 (+ 623us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:25.604772 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.604879 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.604903 (+ 24us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.605690 (+ 787us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.605695 (+ 5us) client_negotiation.cc:770] Sending connection context
0430 08:01:25.605794 (+ 99us) client_negotiation.cc:241] Negotiation successful
0430 08:01:25.605873 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260430 08:01:25.606493 22387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:25.602780 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:53489 (local address 127.13.94.130:36179)
0430 08:01:25.602887 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:25.602891 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:25.602903 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:01:25.602943 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:25.602946 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:25.602995 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:25.603063 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:25.603511 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.604008 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:25.604866 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:25.605051 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:25.605101 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:25.605297 (+ 196us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:25.605453 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:25.606357 (+ 904us) server_negotiation.cc:300] Negotiation successful
0430 08:01:25.606401 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260430 08:01:25.627753 21871 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:25.627938 21871 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:25.630172 21871 catalog_manager.cc:5958] T 00000000000000000000000000000000 P fe04a193dc96440f8486edec4dca94c1: Sending DeleteTablet for 3 replicas of tablet bc0aa0ca63dd4b75a08970b0c9b1827f
I20260430 08:01:25.630776 22010 tablet_service.cc:1558] Processing DeleteTablet for tablet bc0aa0ca63dd4b75a08970b0c9b1827f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:25 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:34850
I20260430 08:01:25.630826 22294 tablet_service.cc:1558] Processing DeleteTablet for tablet bc0aa0ca63dd4b75a08970b0c9b1827f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:25 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:42634
I20260430 08:01:25.630997 22466 tablet_replica.cc:333] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: stopping tablet replica
I20260430 08:01:25.631059 22467 tablet_replica.cc:333] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: stopping tablet replica
I20260430 08:01:25.631107 22466 raft_consensus.cc:2243] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:25.631186 22467 raft_consensus.cc:2243] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:25.631222 22466 raft_consensus.cc:2272] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:25.631304 22467 raft_consensus.cc:2272] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:25.631821 22152 tablet_service.cc:1558] Processing DeleteTablet for tablet bc0aa0ca63dd4b75a08970b0c9b1827f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:25 UTC) from {username='oryx', principal='oryx/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:37040
I20260430 08:01:25.632053 22465 tablet_replica.cc:333] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: stopping tablet replica
I20260430 08:01:25.632169 22465 raft_consensus.cc:2243] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:01:25.632387 22465 raft_consensus.cc:2272] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:25.632428 22467 ts_tablet_manager.cc:1916] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:25.633141 22466 ts_tablet_manager.cc:1916] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:25.635028 22467 ts_tablet_manager.cc:1929] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:01:25.635097 22467 log.cc:1199] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/bc0aa0ca63dd4b75a08970b0c9b1827f
I20260430 08:01:25.635356 22467 ts_tablet_manager.cc:1950] T bc0aa0ca63dd4b75a08970b0c9b1827f P 6f661fe8f5e241acbf8b5482b32be069: Deleting consensus metadata
I20260430 08:01:25.635632 22466 ts_tablet_manager.cc:1929] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:01:25.635701 22466 log.cc:1199] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-0/wal/wals/bc0aa0ca63dd4b75a08970b0c9b1827f
I20260430 08:01:25.635954 22466 ts_tablet_manager.cc:1950] T bc0aa0ca63dd4b75a08970b0c9b1827f P 4bfb7409eeab4884a492e47bb318e059: Deleting consensus metadata
I20260430 08:01:25.636364 22465 ts_tablet_manager.cc:1916] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:25.636848 21855 catalog_manager.cc:5002] TS 6f661fe8f5e241acbf8b5482b32be069 (127.13.94.131:42691): tablet bc0aa0ca63dd4b75a08970b0c9b1827f (table test-table [id=10b268ddea314442a277374a58ac3dcc]) successfully deleted
I20260430 08:01:25.636874 21857 catalog_manager.cc:5002] TS 4bfb7409eeab4884a492e47bb318e059 (127.13.94.129:45127): tablet bc0aa0ca63dd4b75a08970b0c9b1827f (table test-table [id=10b268ddea314442a277374a58ac3dcc]) successfully deleted
I20260430 08:01:25.637683 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 21945
I20260430 08:01:25.639067 22465 ts_tablet_manager.cc:1929] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260430 08:01:25.639140 22465 log.cc:1199] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777536011314967-13690-0/minicluster-data/ts-1/wal/wals/bc0aa0ca63dd4b75a08970b0c9b1827f
I20260430 08:01:25.639387 22465 ts_tablet_manager.cc:1950] T bc0aa0ca63dd4b75a08970b0c9b1827f P 7792d82ea35a4105a892c66260f1b546: Deleting consensus metadata
I20260430 08:01:25.640072 21855 catalog_manager.cc:5002] TS 7792d82ea35a4105a892c66260f1b546 (127.13.94.130:36179): tablet bc0aa0ca63dd4b75a08970b0c9b1827f (table test-table [id=10b268ddea314442a277374a58ac3dcc]) successfully deleted
I20260430 08:01:25.646970 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 22087
I20260430 08:01:25.654471 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 22229
I20260430 08:01:25.661932 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 21839
2026-04-30T08:01:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4530 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:01:25 dist-test-slave-9bxp krb5kdc[22501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:01:25 dist-test-slave-9bxp krb5kdc[22501](info): set up 2 sockets
Apr 30 08:01:25 dist-test-slave-9bxp krb5kdc[22501](info): commencing operation
krb5kdc: starting...
W20260430 08:01:27.674157 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:01:27 dist-test-slave-9bxp krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:01:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:27.820960 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:34377
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:34169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:27.923998 22517 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:27.924324 22517 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:27.924443 22517 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:27.927848 22517 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:27.927955 22517 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:27.928001 22517 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:27.928064 22517 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:27.928119 22517 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:27.932804 22517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:34169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:34377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:27.934005 22517 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:27.934890 22517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:27.940841 22523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:27.940932 22522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:27.940897 22517 server_base.cc:1061] running on GCE node
W20260430 08:01:27.940796 22525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:27.941535 22517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:27.942512 22517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:27.943715 22517 hybrid_clock.cc:648] HybridClock initialized: now 1777536087943687 us; error 37 us; skew 500 ppm
W20260430 08:01:27.947419 22517 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-04-30T08:01:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2284 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:01:28 dist-test-slave-9bxp krb5kdc[22530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:01:28 dist-test-slave-9bxp krb5kdc[22530](info): set up 2 sockets
Apr 30 08:01:28 dist-test-slave-9bxp krb5kdc[22530](info): commencing operation
krb5kdc: starting...
W20260430 08:01:29.938166 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:01:30 dist-test-slave-9bxp krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:01:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:30.095924 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:37101
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:37101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:30.200975 22546 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:30.201232 22546 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:30.201337 22546 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:30.204933 22546 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:30.205008 22546 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:30.205034 22546 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:30.205052 22546 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:30.205097 22546 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:30.209630 22546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:37101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:37101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:30.210814 22546 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:30.211712 22546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:30.218160 22551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:30.218154 22552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:30.218154 22554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:30.218398 22546 server_base.cc:1061] running on GCE node
I20260430 08:01:30.219000 22546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:30.219913 22546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:30.221092 22546 hybrid_clock.cc:648] HybridClock initialized: now 1777536090221073 us; error 40 us; skew 500 ppm
Apr 30 08:01:30 dist-test-slave-9bxp krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536090, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:30.223733 22546 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:01:30.224938 22546 webserver.cc:492] Webserver started at http://127.13.94.190:33529/ using document root <none> and password file <none>
I20260430 08:01:30.225487 22546 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:30.225561 22546 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:30.225776 22546 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:30.227434 22546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "b0ab3547b2d14dd18d966ba881f05786"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "24bf068e7dc7cd02167d062cfe17aba0"
server_key_iv: "5fe94523f045d02aa9d9df428a01cdb5"
server_key_version: "encryptionkey@0"
I20260430 08:01:30.227932 22546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "b0ab3547b2d14dd18d966ba881f05786"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "24bf068e7dc7cd02167d062cfe17aba0"
server_key_iv: "5fe94523f045d02aa9d9df428a01cdb5"
server_key_version: "encryptionkey@0"
I20260430 08:01:30.231305 22546 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260430 08:01:30.233654 22561 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:30.234685 22546 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:30.234829 22546 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "b0ab3547b2d14dd18d966ba881f05786"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "24bf068e7dc7cd02167d062cfe17aba0"
server_key_iv: "5fe94523f045d02aa9d9df428a01cdb5"
server_key_version: "encryptionkey@0"
I20260430 08:01:30.234943 22546 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:30.261293 22546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:30.272253 22546 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:30.272516 22546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:30.281623 22546 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:37101
I20260430 08:01:30.281618 22613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:37101 every 8 connection(s)
I20260430 08:01:30.282755 22546 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:30.285640 22614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:30.291256 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: Bootstrap starting.
I20260430 08:01:30.292688 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 22546
I20260430 08:01:30.292841 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:30.293081 13690 external_mini_cluster.cc:1468] Setting key 0e952ca457ede7283c572c06d43d818a
I20260430 08:01:30.293967 22614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:30.294883 22614 log.cc:826] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:30.296876 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: No bootstrap required, opened a new log
I20260430 08:01:30.299564 22614 raft_consensus.cc:359] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } }
I20260430 08:01:30.299803 22614 raft_consensus.cc:385] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:30.299889 22614 raft_consensus.cc:740] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ab3547b2d14dd18d966ba881f05786, State: Initialized, Role: FOLLOWER
I20260430 08:01:30.300364 22614 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } }
I20260430 08:01:30.300590 22614 raft_consensus.cc:399] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Apr 30 08:01:30 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:30.300668 22614 raft_consensus.cc:493] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:30.300819 22614 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:30.301805 22614 raft_consensus.cc:515] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } }
I20260430 08:01:30.302209 22614 leader_election.cc:304] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ab3547b2d14dd18d966ba881f05786; no voters:
I20260430 08:01:30.302614 22614 leader_election.cc:290] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:30.302798 22619 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:30.303062 22619 raft_consensus.cc:697] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 1 LEADER]: Becoming Leader. State: Replica: b0ab3547b2d14dd18d966ba881f05786, State: Running, Role: LEADER
I20260430 08:01:30.303544 22619 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } }
I20260430 08:01:30.303951 22614 sys_catalog.cc:565] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:30.305051 22620 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ab3547b2d14dd18d966ba881f05786. Latest consensus state: current_term: 1 leader_uuid: "b0ab3547b2d14dd18d966ba881f05786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } } }
I20260430 08:01:30.305285 22620 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:30.305490 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0ab3547b2d14dd18d966ba881f05786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } } }
I20260430 08:01:30.305652 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:30.305810 22628 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:30.307902 22617 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:30.294336 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.13.94.190:37101)
0430 08:01:30.294788 (+ 452us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:30.294798 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:30.294834 (+ 36us) server_negotiation.cc:408] Connection header received
0430 08:01:30.295465 (+ 631us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:30.295483 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:30.295771 (+ 288us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:30.296114 (+ 343us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:30.297153 (+ 1039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.297896 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:30.298590 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.298886 (+ 296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:30.301332 (+ 2446us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:30.301357 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:30.301370 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:30.301404 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:30.304576 (+ 3172us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:30.305291 (+ 715us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:30.305299 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:30.305308 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:30.305423 (+ 115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:30.305768 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:30.305771 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:30.305773 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:30.306233 (+ 460us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:30.306450 (+ 217us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:30.306819 (+ 369us) server_negotiation.cc:300] Negotiation successful
0430 08:01:30.307078 (+ 259us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":145,"threads_started":1}
I20260430 08:01:30.309176 22628 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:30.315377 22628 catalog_manager.cc:1357] Generated new cluster ID: b01ff7000e964905ae0b1dc7e819e507
I20260430 08:01:30.315475 22628 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:30.327271 22628 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:30.328619 22628 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:30.336259 22628 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: Generated new TSK 0
I20260430 08:01:30.337028 22628 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:30.397210 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:37101
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:30.507261 22642 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:30.507505 22642 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:30.507563 22642 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:30.511005 22642 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:30.511081 22642 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:30.511200 22642 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:01:30.515712 22642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:30.516893 22642 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:30.517783 22642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:30.524742 22647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:30.524755 22648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:30.525108 22642 server_base.cc:1061] running on GCE node
W20260430 08:01:30.524760 22650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:30.525519 22642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:30.526122 22642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:30.527279 22642 hybrid_clock.cc:648] HybridClock initialized: now 1777536090527254 us; error 40 us; skew 500 ppm
Apr 30 08:01:30 dist-test-slave-9bxp krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536090, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:30.530206 22642 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:01:30.531419 22642 webserver.cc:492] Webserver started at http://127.13.94.129:42483/ using document root <none> and password file <none>
I20260430 08:01:30.532047 22642 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:30.532127 22642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:30.532343 22642 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:30.534248 22642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "9600251973fd4670851ccf27ef1adef7"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "58403eff48871fbfb3f9118272dd5cf9"
server_key_iv: "3bbbdc7f37fe12dc16539bbe502d3c0b"
server_key_version: "encryptionkey@0"
I20260430 08:01:30.534797 22642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "9600251973fd4670851ccf27ef1adef7"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "58403eff48871fbfb3f9118272dd5cf9"
server_key_iv: "3bbbdc7f37fe12dc16539bbe502d3c0b"
server_key_version: "encryptionkey@0"
I20260430 08:01:30.538519 22642 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260430 08:01:30.540990 22657 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:30.542181 22642 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:01:30.542338 22642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "9600251973fd4670851ccf27ef1adef7"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "58403eff48871fbfb3f9118272dd5cf9"
server_key_iv: "3bbbdc7f37fe12dc16539bbe502d3c0b"
server_key_version: "encryptionkey@0"
I20260430 08:01:30.542459 22642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:30.563325 22642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:30.566537 22642 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:30.566766 22642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:30.567380 22642 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:30.568320 22642 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:30.568394 22642 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:30.568466 22642 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:30.568521 22642 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:30.578876 22642 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:35159
I20260430 08:01:30.578890 22770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:35159 every 8 connection(s)
I20260430 08:01:30.579907 22642 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:01:30.583945 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 22642
I20260430 08:01:30.584107 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:01:30.584396 13690 external_mini_cluster.cc:1468] Setting key 726a14d562ad359599d33ba858f776d3
Apr 30 08:01:30 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536090, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
I20260430 08:01:30.594215 22617 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:30.582014 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:34509 (local address 127.13.94.190:37101)
0430 08:01:30.582177 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:30.582180 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:30.582846 (+ 666us) server_negotiation.cc:408] Connection header received
0430 08:01:30.583717 (+ 871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:30.583722 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:30.583789 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:30.583893 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:30.585918 (+ 2025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.586424 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:30.587115 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.587297 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:30.590027 (+ 2730us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:30.590060 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:30.590065 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:30.590103 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:30.591971 (+ 1868us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:30.592497 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:30.592501 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:30.592503 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:30.592565 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:30.593039 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:30.593044 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:30.593046 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:30.593260 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:30.593380 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:30.593887 (+ 507us) server_negotiation.cc:300] Negotiation successful
0430 08:01:30.594024 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260430 08:01:30.594775 22773 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:30.582205 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:34509)
0430 08:01:30.582684 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:30.582720 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:30.583504 (+ 784us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:30.584045 (+ 541us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:30.584054 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:30.584657 (+ 603us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:30.585765 (+ 1108us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:30.585782 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.586555 (+ 773us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:30.586559 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:30.586987 (+ 428us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:30.586994 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.587215 (+ 221us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:30.587884 (+ 669us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:30.587905 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:30.589791 (+ 1886us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:30.592118 (+ 2327us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:30.592124 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:30.592135 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:30.592383 (+ 248us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:30.592670 (+ 287us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:30.592673 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:30.592674 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:30.592938 (+ 264us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:30.593378 (+ 440us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:30.593383 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:30.593619 (+ 236us) client_negotiation.cc:770] Sending connection context
0430 08:01:30.593806 (+ 187us) client_negotiation.cc:241] Negotiation successful
0430 08:01:30.594001 (+ 195us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":100,"threads_started":1}
I20260430 08:01:30.596232 22771 heartbeater.cc:344] Connected to a master server at 127.13.94.190:37101
I20260430 08:01:30.596577 22771 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
I20260430 08:01:30.597334 22771 heartbeater.cc:507] Master 127.13.94.190:37101 requested a full tablet report, sending...
I20260430 08:01:30.598979 22578 ts_manager.cc:194] Registered new tserver with Master: 9600251973fd4670851ccf27ef1adef7 (127.13.94.129:35159)
I20260430 08:01:30.600183 22578 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:34509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:30.636047 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:37101
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:30.742684 22778 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:30.742915 22778 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:30.743010 22778 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:30.746563 22778 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:30.746665 22778 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:30.746824 22778 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:01:30.752187 22778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:30.753484 22778 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:30.754385 22778 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:30.760882 22783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:30.760879 22786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:30.760881 22784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:30.761252 22778 server_base.cc:1061] running on GCE node
I20260430 08:01:30.761647 22778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:30.762287 22778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:30.763505 22778 hybrid_clock.cc:648] HybridClock initialized: now 1777536090763478 us; error 33 us; skew 500 ppm
Apr 30 08:01:30 dist-test-slave-9bxp krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536090, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:30.766351 22778 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:01:30.767467 22778 webserver.cc:492] Webserver started at http://127.13.94.130:39095/ using document root <none> and password file <none>
I20260430 08:01:30.768087 22778 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:30.768165 22778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:30.768385 22778 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:30.770212 22778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "c4b862ae0a6d427ba376b9a7e7c874f2"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "8592f7da7b374cc11935d7d09725a4f4"
server_key_iv: "9de3ca2eba66079d1015bc3869c011fd"
server_key_version: "encryptionkey@0"
I20260430 08:01:30.770725 22778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "c4b862ae0a6d427ba376b9a7e7c874f2"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "8592f7da7b374cc11935d7d09725a4f4"
server_key_iv: "9de3ca2eba66079d1015bc3869c011fd"
server_key_version: "encryptionkey@0"
I20260430 08:01:30.774091 22778 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 08:01:30.776268 22793 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:30.777329 22778 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:30.777472 22778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "c4b862ae0a6d427ba376b9a7e7c874f2"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "8592f7da7b374cc11935d7d09725a4f4"
server_key_iv: "9de3ca2eba66079d1015bc3869c011fd"
server_key_version: "encryptionkey@0"
I20260430 08:01:30.777585 22778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:30.798712 22778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:30.801784 22778 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:30.802007 22778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:30.802621 22778 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:30.803577 22778 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:30.803653 22778 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:30.803725 22778 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:30.803778 22778 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:30.814694 22778 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:36099
I20260430 08:01:30.814762 22906 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:36099 every 8 connection(s)
I20260430 08:01:30.815757 22778 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:01:30.822317 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 22778
I20260430 08:01:30.822455 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:01:30.822808 13690 external_mini_cluster.cc:1468] Setting key afb8ddf0511d66eb331ffdfabd0f8ede
Apr 30 08:01:30 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536090, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:30.829140 22617 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:30.817714 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:49283 (local address 127.13.94.190:37101)
0430 08:01:30.817927 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:30.817931 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:30.818564 (+ 633us) server_negotiation.cc:408] Connection header received
0430 08:01:30.819474 (+ 910us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:30.819478 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:30.819540 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:30.819655 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:30.821288 (+ 1633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.821799 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:30.822495 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.822710 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:30.825343 (+ 2633us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:30.825361 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:30.825363 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:30.825391 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:30.826947 (+ 1556us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:30.827542 (+ 595us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:30.827545 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:30.827546 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:30.827611 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:30.827964 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:30.827967 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:30.827968 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:30.828114 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:30.828190 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:30.828907 (+ 717us) server_negotiation.cc:300] Negotiation successful
0430 08:01:30.829011 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260430 08:01:30.830178 22909 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:30.817935 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:49283)
0430 08:01:30.818374 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:30.818408 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:30.819251 (+ 843us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:30.819821 (+ 570us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:30.819835 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:30.820210 (+ 375us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:30.821136 (+ 926us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:30.821150 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.821929 (+ 779us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:30.821934 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:30.822326 (+ 392us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:30.822332 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:30.822568 (+ 236us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:30.823192 (+ 624us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:30.823222 (+ 30us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:30.825155 (+ 1933us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:30.827053 (+ 1898us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:30.827064 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:30.827080 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:30.827440 (+ 360us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:30.827705 (+ 265us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:30.827712 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:30.827717 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:30.827870 (+ 153us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:30.828229 (+ 359us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:30.828239 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:30.828577 (+ 338us) client_negotiation.cc:770] Sending connection context
0430 08:01:30.828878 (+ 301us) client_negotiation.cc:241] Negotiation successful
0430 08:01:30.829162 (+ 284us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":95,"threads_started":1}
I20260430 08:01:30.831583 22907 heartbeater.cc:344] Connected to a master server at 127.13.94.190:37101
I20260430 08:01:30.831873 22907 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:30.832412 22907 heartbeater.cc:507] Master 127.13.94.190:37101 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
I20260430 08:01:30.833760 22578 ts_manager.cc:194] Registered new tserver with Master: c4b862ae0a6d427ba376b9a7e7c874f2 (127.13.94.130:36099)
I20260430 08:01:30.834393 22578 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:49283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:30.880523 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:37101
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:30.989537 22914 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:30.989789 22914 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:30.989857 22914 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:30.993273 22914 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:30.993350 22914 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:30.993431 22914 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:01:30.998370 22914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:30.999557 22914 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:31.000398 22914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:31.007345 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:31.007529 22914 server_base.cc:1061] running on GCE node
W20260430 08:01:31.007445 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:31.007336 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:31.008062 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:31.008668 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:31.009904 22914 hybrid_clock.cc:648] HybridClock initialized: now 1777536091009889 us; error 28 us; skew 500 ppm
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536091, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:31.012949 22914 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:01:31.014062 22914 webserver.cc:492] Webserver started at http://127.13.94.131:37719/ using document root <none> and password file <none>
I20260430 08:01:31.014605 22914 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:31.014652 22914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:31.014812 22914 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:31.016673 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "26b32891dffe4a02a0732dc2c6434b5d"
format_stamp: "Formatted at 2026-04-30 08:01:31 on dist-test-slave-9bxp"
server_key: "d8710ed83ee59c55593b79dcbf250958"
server_key_iv: "82368479732bf519672cb10e8ad70568"
server_key_version: "encryptionkey@0"
I20260430 08:01:31.017196 22914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "26b32891dffe4a02a0732dc2c6434b5d"
format_stamp: "Formatted at 2026-04-30 08:01:31 on dist-test-slave-9bxp"
server_key: "d8710ed83ee59c55593b79dcbf250958"
server_key_iv: "82368479732bf519672cb10e8ad70568"
server_key_version: "encryptionkey@0"
I20260430 08:01:31.020545 22914 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:01:31.022821 22929 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.023872 22914 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:31.024010 22914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "26b32891dffe4a02a0732dc2c6434b5d"
format_stamp: "Formatted at 2026-04-30 08:01:31 on dist-test-slave-9bxp"
server_key: "d8710ed83ee59c55593b79dcbf250958"
server_key_iv: "82368479732bf519672cb10e8ad70568"
server_key_version: "encryptionkey@0"
I20260430 08:01:31.024122 22914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:31.036049 22914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:31.039712 22914 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:31.039916 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:31.040529 22914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:31.041523 22914 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:31.041581 22914 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.041632 22914 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:31.041689 22914 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.051131 22914 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:34637
I20260430 08:01:31.051167 23042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:34637 every 8 connection(s)
I20260430 08:01:31.052172 22914 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:01:31.057080 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 22914
I20260430 08:01:31.057190 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:01:31.057436 13690 external_mini_cluster.cc:1468] Setting key f25b24f214cfb67f731153f6950f2372
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536091, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:31.065554 22617 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.054273 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:54255 (local address 127.13.94.190:37101)
0430 08:01:31.054459 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.054463 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.055350 (+ 887us) server_negotiation.cc:408] Connection header received
0430 08:01:31.056243 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.056246 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.056298 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.056379 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.058288 (+ 1909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.058929 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.059662 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.059880 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.062186 (+ 2306us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:31.062203 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:31.062206 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:31.062235 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:31.063743 (+ 1508us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:31.064223 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:31.064226 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:31.064228 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:31.064272 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:31.064547 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:31.064549 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:31.064550 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:31.064760 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:31.064842 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:31.065322 (+ 480us) server_negotiation.cc:300] Negotiation successful
0430 08:01:31.065429 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260430 08:01:31.066241 23045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.054568 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:54255)
0430 08:01:31.055192 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.055230 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.056062 (+ 832us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.056564 (+ 502us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.056576 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.057108 (+ 532us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.058144 (+ 1036us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.058160 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.059082 (+ 922us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.059086 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.059515 (+ 429us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.059526 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.059753 (+ 227us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.060355 (+ 602us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.060376 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.062026 (+ 1650us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:31.063879 (+ 1853us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:31.063885 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:31.063896 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:31.064132 (+ 236us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:31.064351 (+ 219us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:31.064354 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:31.064355 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:31.064470 (+ 115us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:31.064867 (+ 397us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:31.064872 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:31.065098 (+ 226us) client_negotiation.cc:770] Sending connection context
0430 08:01:31.065280 (+ 182us) client_negotiation.cc:241] Negotiation successful
0430 08:01:31.065476 (+ 196us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":102,"threads_started":1}
I20260430 08:01:31.067490 23043 heartbeater.cc:344] Connected to a master server at 127.13.94.190:37101
I20260430 08:01:31.067780 23043 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:31.068349 23043 heartbeater.cc:507] Master 127.13.94.190:37101 requested a full tablet report, sending...
I20260430 08:01:31.069481 22578 ts_manager.cc:194] Registered new tserver with Master: 26b32891dffe4a02a0732dc2c6434b5d (127.13.94.131:34637)
I20260430 08:01:31.070057 22578 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:54255
I20260430 08:01:31.071730 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.129.keytab.
Entry for principal oryx/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.129.keytab.
WARNING: no policy specified for oryx/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.130.keytab.
Entry for principal oryx/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.130.keytab.
WARNING: no policy specified for oryx/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.131.keytab.
Entry for principal oryx/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.131.keytab.
I20260430 08:01:31.152390 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 22642
I20260430 08:01:31.160477 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 22778
I20260430 08:01:31.166790 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 22914
I20260430 08:01:31.172940 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 22546
I20260430 08:01:31.179720 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:37101
--webserver_interface=127.13.94.190
--webserver_port=33529
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:37101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:31.286567 23052 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:31.286867 23052 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:31.286959 23052 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:31.290380 23052 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:31.290496 23052 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:31.290541 23052 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:31.290611 23052 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:31.290670 23052 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:31.295176 23052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:37101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:37101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=33529
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:31.296365 23052 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:31.297545 23052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:31.303702 23057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:31.303701 23058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:31.303725 23060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:31.304291 23052 server_base.cc:1061] running on GCE node
I20260430 08:01:31.304893 23052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:31.305920 23052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:31.307075 23052 hybrid_clock.cc:648] HybridClock initialized: now 1777536091307050 us; error 46 us; skew 500 ppm
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536091, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:31.310003 23052 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:01:31.311172 23052 webserver.cc:492] Webserver started at http://127.13.94.190:33529/ using document root <none> and password file <none>
I20260430 08:01:31.311884 23052 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:31.311950 23052 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:31.316036 23052 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260430 08:01:31.318305 23067 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.319389 23052 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:01:31.319530 23052 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "b0ab3547b2d14dd18d966ba881f05786"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "24bf068e7dc7cd02167d062cfe17aba0"
server_key_iv: "5fe94523f045d02aa9d9df428a01cdb5"
server_key_version: "encryptionkey@0"
I20260430 08:01:31.320001 23052 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:31.331439 23052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:31.334352 23052 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:31.334554 23052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:31.342883 23052 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:37101
I20260430 08:01:31.342900 23119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:37101 every 8 connection(s)
I20260430 08:01:31.344035 23052 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:31.345914 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 23052
I20260430 08:01:31.346351 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:35159
--local_ip_for_outbound_sockets=127.13.94.129
--tserver_master_addrs=127.13.94.190:37101
--webserver_port=42483
--webserver_interface=127.13.94.129
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.94.129
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260430 08:01:31.348873 23120 sys_catalog.cc:263] Verifying existing consensus state
I20260430 08:01:31.350551 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: Bootstrap starting.
I20260430 08:01:31.367000 23120 log.cc:826] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:31.374053 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 08:01:31.374480 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: Bootstrap complete.
I20260430 08:01:31.378113 23120 raft_consensus.cc:359] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } }
I20260430 08:01:31.378473 23120 raft_consensus.cc:740] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ab3547b2d14dd18d966ba881f05786, State: Initialized, Role: FOLLOWER
I20260430 08:01:31.378940 23120 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } }
I20260430 08:01:31.379082 23120 raft_consensus.cc:399] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:31.379168 23120 raft_consensus.cc:493] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:31.379285 23120 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 1 FOLLOWER]: Advancing to term 2
I20260430 08:01:31.380937 23120 raft_consensus.cc:515] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } }
I20260430 08:01:31.381269 23120 leader_election.cc:304] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ab3547b2d14dd18d966ba881f05786; no voters:
I20260430 08:01:31.381634 23120 leader_election.cc:290] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260430 08:01:31.381695 23124 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 08:01:31.381891 23124 raft_consensus.cc:697] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [term 2 LEADER]: Becoming Leader. State: Replica: b0ab3547b2d14dd18d966ba881f05786, State: Running, Role: LEADER
I20260430 08:01:31.382170 23124 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } }
I20260430 08:01:31.382788 23120 sys_catalog.cc:565] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:31.383406 23125 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ab3547b2d14dd18d966ba881f05786. Latest consensus state: current_term: 2 leader_uuid: "b0ab3547b2d14dd18d966ba881f05786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } } }
I20260430 08:01:31.383519 23125 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:31.384097 23126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b0ab3547b2d14dd18d966ba881f05786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ab3547b2d14dd18d966ba881f05786" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 37101 } } }
I20260430 08:01:31.384238 23126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:31.384478 23129 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:31.387676 23129 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:31.388686 23129 catalog_manager.cc:1269] Loaded cluster ID: b01ff7000e964905ae0b1dc7e819e507
I20260430 08:01:31.388811 23129 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:31.391283 23129 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:31.392069 23129 catalog_manager.cc:6055] T 00000000000000000000000000000000 P b0ab3547b2d14dd18d966ba881f05786: Loaded TSK: 0
I20260430 08:01:31.393076 23129 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260430 08:01:31.457373 23122 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:31.457620 23122 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:31.457732 23122 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:31.461122 23122 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:31.461195 23122 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:31.461347 23122 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:01:31.465726 23122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.129.keytab
--principal=oryx/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:35159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=42483
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:31.466938 23122 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:31.467846 23122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:31.474265 23149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:31.474320 23146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:31.474331 23147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:31.474628 23122 server_base.cc:1061] running on GCE node
I20260430 08:01:31.475062 23122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:31.475610 23122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:31.476780 23122 hybrid_clock.cc:648] HybridClock initialized: now 1777536091476754 us; error 41 us; skew 500 ppm
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536091, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:31.479369 23122 init.cc:377] Logged in from keytab as oryx/127.13.94.129@KRBTEST.COM (short username oryx)
I20260430 08:01:31.480496 23122 webserver.cc:492] Webserver started at http://127.13.94.129:42483/ using document root <none> and password file <none>
I20260430 08:01:31.481127 23122 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:31.481216 23122 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:31.484608 23122 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:31.486596 23156 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.487659 23122 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:31.487773 23122 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "9600251973fd4670851ccf27ef1adef7"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "58403eff48871fbfb3f9118272dd5cf9"
server_key_iv: "3bbbdc7f37fe12dc16539bbe502d3c0b"
server_key_version: "encryptionkey@0"
I20260430 08:01:31.488202 23122 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:31.528631 23122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:31.531649 23122 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:31.531864 23122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:31.532522 23122 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:31.533448 23122 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:31.533496 23122 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.533535 23122 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:31.533592 23122 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.545394 23122 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:35159
I20260430 08:01:31.545454 23269 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:35159 every 8 connection(s)
I20260430 08:01:31.546499 23122 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:01:31.553046 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 23122
I20260430 08:01:31.553583 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:36099
--local_ip_for_outbound_sockets=127.13.94.130
--tserver_master_addrs=127.13.94.190:37101
--webserver_port=39095
--webserver_interface=127.13.94.130
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.94.130
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:31.558624 23272 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.548800 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:56323)
0430 08:01:31.549242 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.549285 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.550118 (+ 833us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.551341 (+ 1223us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.551349 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.551730 (+ 381us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.552564 (+ 834us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.552577 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.553990 (+ 1413us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.553993 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.554403 (+ 410us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.554409 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.554645 (+ 236us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.555323 (+ 678us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.555342 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.557733 (+ 2391us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":131,"threads_started":1}
W20260430 08:01:31.558600 23273 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:31.548745 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:56323 (local address 127.13.94.190:37101)
0430 08:01:31.549346 (+ 601us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.549357 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.549400 (+ 43us) server_negotiation.cc:408] Connection header received
0430 08:01:31.550530 (+ 1130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.550545 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.550916 (+ 371us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.551309 (+ 393us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.552859 (+ 1550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.553811 (+ 952us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.554536 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.554792 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.557888 (+ 3096us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:56323: BlockingRecv error: recv got EOF from 127.13.94.129:56323 (error 108)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":138,"threads_started":1}
W20260430 08:01:31.559453 23270 heartbeater.cc:646] Failed to heartbeat to 127.13.94.190:37101 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.94.190:37101: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:31.565351 23272 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.559933 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:39753)
0430 08:01:31.560279 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.560293 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.560372 (+ 79us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.560783 (+ 411us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.560786 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.560965 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.561159 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.561165 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.562042 (+ 877us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.562050 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.562449 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.562455 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.562591 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.563193 (+ 602us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.563207 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.565184 (+ 1977us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":130}
W20260430 08:01:31.565415 23273 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:31.560137 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:39753 (local address 127.13.94.190:37101)
0430 08:01:31.560516 (+ 379us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.560519 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.560530 (+ 11us) server_negotiation.cc:408] Connection header received
0430 08:01:31.560583 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.560586 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.560627 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.560762 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.561315 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.561884 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.562573 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.562783 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.565303 (+ 2520us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:39753: BlockingRecv error: recv got EOF from 127.13.94.129:39753 (error 108)
Metrics: {"server-negotiator.queue_time_us":309}
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:31.570739 23272 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.566211 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:38737)
0430 08:01:31.566381 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.566394 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.566487 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.566751 (+ 264us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.566754 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.566916 (+ 162us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.567073 (+ 157us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.567078 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.567774 (+ 696us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.567779 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.568203 (+ 424us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.568211 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.568340 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.568880 (+ 540us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.568892 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.570579 (+ 1687us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260430 08:01:31.570796 23273 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:31.566262 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:38737 (local address 127.13.94.190:37101)
0430 08:01:31.566381 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.566387 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.566440 (+ 53us) server_negotiation.cc:408] Connection header received
0430 08:01:31.566604 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.566607 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.566648 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.566733 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.567189 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.567666 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.568325 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.568527 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.570690 (+ 2163us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:38737: BlockingRecv error: recv got EOF from 127.13.94.129:38737 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260430 08:01:31.571220 23270 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260430 08:01:31.665030 23274 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:31.665354 23274 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:31.665459 23274 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:31.668691 23274 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:31.668814 23274 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:31.668939 23274 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:01:31.673301 23274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.130.keytab
--principal=oryx/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:36099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=39095
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:31.674650 23274 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:31.675529 23274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:31.681857 23279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:31.681821 23282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:31.681860 23280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:31.682144 23274 server_base.cc:1061] running on GCE node
I20260430 08:01:31.682611 23274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:31.683199 23274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:31.684371 23274 hybrid_clock.cc:648] HybridClock initialized: now 1777536091684352 us; error 40 us; skew 500 ppm
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536091, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:31.687062 23274 init.cc:377] Logged in from keytab as oryx/127.13.94.130@KRBTEST.COM (short username oryx)
I20260430 08:01:31.688145 23274 webserver.cc:492] Webserver started at http://127.13.94.130:39095/ using document root <none> and password file <none>
I20260430 08:01:31.688782 23274 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:31.688868 23274 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:31.692546 23274 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:31.694725 23289 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.695779 23274 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:31.695904 23274 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "c4b862ae0a6d427ba376b9a7e7c874f2"
format_stamp: "Formatted at 2026-04-30 08:01:30 on dist-test-slave-9bxp"
server_key: "8592f7da7b374cc11935d7d09725a4f4"
server_key_iv: "9de3ca2eba66079d1015bc3869c011fd"
server_key_version: "encryptionkey@0"
I20260430 08:01:31.696343 23274 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:31.713572 23274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:31.716233 23274 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:31.716441 23274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:31.717090 23274 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:31.717993 23274 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:31.718041 23274 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.718080 23274 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:31.718138 23274 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.728138 23274 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:36099
I20260430 08:01:31.728168 23402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:36099 every 8 connection(s)
I20260430 08:01:31.729255 23274 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:01:31.730856 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 23274
I20260430 08:01:31.731356 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:34637
--local_ip_for_outbound_sockets=127.13.94.131
--tserver_master_addrs=127.13.94.190:37101
--webserver_port=37719
--webserver_interface=127.13.94.131
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.94.131
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
W20260430 08:01:31.739864 23273 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:31.731048 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:59001 (local address 127.13.94.190:37101)
0430 08:01:31.731184 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.731188 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.732008 (+ 820us) server_negotiation.cc:408] Connection header received
0430 08:01:31.732971 (+ 963us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.732975 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.733047 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.733156 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.734694 (+ 1538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.735206 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.735891 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.736070 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.739733 (+ 3663us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:59001: BlockingRecv error: recv got EOF from 127.13.94.130:59001 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260430 08:01:31.740397 23405 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.731419 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:59001)
0430 08:01:31.731866 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.731901 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.732759 (+ 858us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.733322 (+ 563us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.733331 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.733696 (+ 365us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.734530 (+ 834us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.734543 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.735334 (+ 791us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.735340 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.735759 (+ 419us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.735765 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.735990 (+ 225us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.736718 (+ 728us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.736763 (+ 45us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.739608 (+ 2845us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
W20260430 08:01:31.740950 23403 heartbeater.cc:646] Failed to heartbeat to 127.13.94.190:37101 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.94.190:37101: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:31.746124 23405 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.741408 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:55057)
0430 08:01:31.741676 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.741692 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.741783 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.742077 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.742080 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.742245 (+ 165us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.742424 (+ 179us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.742429 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.743137 (+ 708us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.743141 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.743510 (+ 369us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.743516 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.743639 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.744197 (+ 558us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.744210 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.745988 (+ 1778us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188}
W20260430 08:01:31.746215 23273 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:31.741541 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:55057 (local address 127.13.94.190:37101)
0430 08:01:31.741661 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.741665 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.741736 (+ 71us) server_negotiation.cc:408] Connection header received
0430 08:01:31.741921 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.741924 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.741973 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.742059 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.742540 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.743027 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.743624 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.743825 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.746097 (+ 2272us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:55057: BlockingRecv error: recv got EOF from 127.13.94.130:55057 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:31.752306 23405 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.746958 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:41553)
0430 08:01:31.747126 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.747141 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.747222 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.747455 (+ 233us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.747458 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.747610 (+ 152us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.747814 (+ 204us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.747820 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.748594 (+ 774us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.748598 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.749217 (+ 619us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.749227 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.749340 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.749864 (+ 524us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.749879 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.752182 (+ 2303us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260430 08:01:31.752743 23403 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260430 08:01:31.752799 23273 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:31.747089 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:41553 (local address 127.13.94.190:37101)
0430 08:01:31.747221 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.747224 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.747239 (+ 15us) server_negotiation.cc:408] Connection header received
0430 08:01:31.747303 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.747306 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.747350 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.747423 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.747951 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.748475 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.749353 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.749528 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.752386 (+ 2858us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:41553: BlockingRecv error: recv got EOF from 127.13.94.130:41553 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260430 08:01:31.837706 23406 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:31.837955 23406 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:31.838070 23406 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:31.841581 23406 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:31.841696 23406 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:31.841818 23406 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:01:31.846140 23406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/krb5kdc/oryx_127.13.94.131.keytab
--principal=oryx/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:34637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=37719
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:37101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:31.847325 23406 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:31.848229 23406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:31.854923 23412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:31.854955 23411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:31.854923 23414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:31.855219 23406 server_base.cc:1061] running on GCE node
I20260430 08:01:31.855713 23406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:31.856328 23406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:31.857535 23406 hybrid_clock.cc:648] HybridClock initialized: now 1777536091857453 us; error 96 us; skew 500 ppm
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536091, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:31.860363 23406 init.cc:377] Logged in from keytab as oryx/127.13.94.131@KRBTEST.COM (short username oryx)
I20260430 08:01:31.861636 23406 webserver.cc:492] Webserver started at http://127.13.94.131:37719/ using document root <none> and password file <none>
I20260430 08:01:31.862246 23406 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:31.862340 23406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:31.866076 23406 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260430 08:01:31.868036 23421 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.869112 23406 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:31.869246 23406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "26b32891dffe4a02a0732dc2c6434b5d"
format_stamp: "Formatted at 2026-04-30 08:01:31 on dist-test-slave-9bxp"
server_key: "d8710ed83ee59c55593b79dcbf250958"
server_key_iv: "82368479732bf519672cb10e8ad70568"
server_key_version: "encryptionkey@0"
I20260430 08:01:31.869678 23406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:31.886410 23406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:31.889592 23406 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:31.889810 23406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:31.890427 23406 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:31.891343 23406 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:31.891417 23406 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.891489 23406 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:31.891543 23406 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:31.901196 23406 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:34637
I20260430 08:01:31.901209 23534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:34637 every 8 connection(s)
I20260430 08:01:31.902366 23406 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:01:31.907544 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 23406
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
W20260430 08:01:31.912623 23273 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:31.904106 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:35113 (local address 127.13.94.190:37101)
0430 08:01:31.904246 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.904250 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.905047 (+ 797us) server_negotiation.cc:408] Connection header received
0430 08:01:31.905895 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.905899 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.905956 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.906067 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.907595 (+ 1528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.908241 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.909106 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.909340 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.912471 (+ 3131us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:35113: BlockingRecv error: recv got EOF from 127.13.94.131:35113 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260430 08:01:31.913038 23537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.904409 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:35113)
0430 08:01:31.904878 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.904912 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.905702 (+ 790us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.906230 (+ 528us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.906238 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.906593 (+ 355us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.907443 (+ 850us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.907457 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.908392 (+ 935us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.908398 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.908985 (+ 587us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.908994 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.909162 (+ 168us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.909742 (+ 580us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.909760 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.912303 (+ 2543us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":94,"threads_started":1}
W20260430 08:01:31.913555 23535 heartbeater.cc:646] Failed to heartbeat to 127.13.94.190:37101 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.94.190:37101: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
I20260430 08:01:31.917352 23539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.908462 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.13.94.190:37101)
0430 08:01:31.908753 (+ 291us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.908757 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.908770 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:31.908805 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.908808 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.908857 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.908954 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.909917 (+ 963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.910630 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.911477 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.911664 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.912933 (+ 1269us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:31.912948 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:31.912963 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:31.912992 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:31.914899 (+ 1907us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:31.915398 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:31.915405 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:31.915410 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:31.915511 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:31.915947 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:31.915951 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:31.915953 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:31.916325 (+ 372us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:31.916536 (+ 211us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:31.916953 (+ 417us) server_negotiation.cc:300] Negotiation successful
0430 08:01:31.917225 (+ 272us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"spinlock_wait_cycles":37504,"thread_start_us":85,"threads_started":1}
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:31.919446 23537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.914058 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:57441)
0430 08:01:31.914178 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.914195 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.914273 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.914516 (+ 243us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.914519 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.914670 (+ 151us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.914849 (+ 179us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.914854 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.915535 (+ 681us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.915540 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.915918 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.915924 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.916049 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.916551 (+ 502us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.916568 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.919329 (+ 2761us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260430 08:01:31.919565 23273 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:31.914136 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:57441 (local address 127.13.94.190:37101)
0430 08:01:31.914253 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.914257 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.914269 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:01:31.914362 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.914365 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.914411 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.914500 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.914968 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.915428 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.916033 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.916241 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.919463 (+ 3222us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:57441: BlockingRecv error: recv got EOF from 127.13.94.131:57441 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:31 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:31.924441 23540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:31.919974 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:38399)
0430 08:01:31.920266 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:31.920282 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:31.920368 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:31.920560 (+ 192us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:31.920562 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:31.920782 (+ 220us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:31.921000 (+ 218us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.921006 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.921682 (+ 676us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.921685 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:31.922084 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:31.922091 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.922179 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.922646 (+ 467us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:31.922658 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:31.924303 (+ 1645us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":118,"threads_started":1}
W20260430 08:01:31.924508 23273 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:31.920050 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:38399 (local address 127.13.94.190:37101)
0430 08:01:31.920185 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:31.920190 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:31.920338 (+ 148us) server_negotiation.cc:408] Connection header received
0430 08:01:31.920429 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:31.920431 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:31.920468 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:31.920530 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:31.921110 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.921577 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:31.922179 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:31.922393 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:31.924420 (+ 2027us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:38399: BlockingRecv error: recv got EOF from 127.13.94.131:38399 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260430 08:01:31.924863 23535 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Apr 30 08:01:32 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:32 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:32.577397 23541 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:32.571987 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:54887)
0430 08:01:32.572319 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:32.572334 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:32.572457 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:32.572816 (+ 359us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:32.572822 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:32.573029 (+ 207us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:32.573259 (+ 230us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:32.573265 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.574179 (+ 914us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:32.574183 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:32.574652 (+ 469us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:32.574659 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.574769 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:32.575338 (+ 569us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:32.575351 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:32.577269 (+ 1918us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
W20260430 08:01:32.577557 23542 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:32.572038 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:54887 (local address 127.13.94.190:37101)
0430 08:01:32.572327 (+ 289us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:32.572331 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:32.572417 (+ 86us) server_negotiation.cc:408] Connection header received
0430 08:01:32.572553 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:32.572558 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:32.572623 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:32.572777 (+ 154us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:32.573381 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.574043 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:32.574800 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.574949 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:32.577410 (+ 2461us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:54887: BlockingRecv error: recv got EOF from 127.13.94.129:54887 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":95,"threads_started":1}
Apr 30 08:01:32 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:32 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:32.760573 23543 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:32.753580 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:35339)
0430 08:01:32.753870 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:32.753887 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:32.754101 (+ 214us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:32.754606 (+ 505us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:32.754611 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:32.754965 (+ 354us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:32.755280 (+ 315us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:32.755290 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.756435 (+ 1145us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:32.756440 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:32.757126 (+ 686us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:32.757136 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.757307 (+ 171us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:32.758055 (+ 748us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:32.758073 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:32.760340 (+ 2267us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
W20260430 08:01:32.760632 23542 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:32.753680 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:35339 (local address 127.13.94.190:37101)
0430 08:01:32.753814 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:32.753818 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:32.753990 (+ 172us) server_negotiation.cc:408] Connection header received
0430 08:01:32.754239 (+ 249us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:32.754242 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:32.754308 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:32.754382 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:32.755439 (+ 1057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.756310 (+ 871us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:32.757283 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.757515 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:32.760498 (+ 2983us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:35339: BlockingRecv error: recv got EOF from 127.13.94.130:35339 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Apr 30 08:01:32 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:32 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:32.930680 23544 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:32.925589 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:47955)
0430 08:01:32.925892 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:32.925906 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:32.926014 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:32.926301 (+ 287us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:32.926304 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:32.926512 (+ 208us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:32.926723 (+ 211us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:32.926729 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.927501 (+ 772us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:32.927504 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:32.927914 (+ 410us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:32.927920 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.928021 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:32.928593 (+ 572us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:32.928608 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:32.930550 (+ 1942us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":138,"threads_started":1}
W20260430 08:01:32.930816 23542 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:32.925683 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:47955 (local address 127.13.94.190:37101)
0430 08:01:32.925828 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:32.925832 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:32.925984 (+ 152us) server_negotiation.cc:408] Connection header received
0430 08:01:32.926097 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:32.926099 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:32.926149 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:32.926222 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:32.926881 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.927396 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:32.928014 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:32.928154 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:32.930684 (+ 2530us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:47955: BlockingRecv error: recv got EOF from 127.13.94.131:47955 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Apr 30 08:01:33 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:33 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:33.583529 23545 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:33.578460 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:39755)
0430 08:01:33.578758 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:33.578774 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:33.578895 (+ 121us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:33.579212 (+ 317us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:33.579215 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:33.579405 (+ 190us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:33.579601 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:33.579606 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.580366 (+ 760us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:33.580369 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:33.580783 (+ 414us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:33.580790 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.580880 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:33.581440 (+ 560us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:33.581453 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:33.583351 (+ 1898us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
W20260430 08:01:33.583595 23546 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:33.578518 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:39755 (local address 127.13.94.190:37101)
0430 08:01:33.578770 (+ 252us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:33.578773 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:33.578894 (+ 121us) server_negotiation.cc:408] Connection header received
0430 08:01:33.578983 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:33.578989 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:33.579069 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:33.579172 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:33.579707 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.580264 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:33.580888 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.581048 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:33.583477 (+ 2429us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:39755: BlockingRecv error: recv got EOF from 127.13.94.129:39755 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":62,"threads_started":1}
Apr 30 08:01:33 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:33 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:33.767243 23547 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:33.761633 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:54529)
0430 08:01:33.761949 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:33.761967 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:33.762090 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:33.762625 (+ 535us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:33.762629 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:33.762837 (+ 208us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:33.763058 (+ 221us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:33.763064 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.763880 (+ 816us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:33.763884 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:33.764290 (+ 406us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:33.764297 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.764402 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:33.765020 (+ 618us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:33.765036 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:33.767112 (+ 2076us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
W20260430 08:01:33.767405 23546 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:33.761714 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:54529 (local address 127.13.94.190:37101)
0430 08:01:33.762364 (+ 650us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:33.762372 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:33.762390 (+ 18us) server_negotiation.cc:408] Connection header received
0430 08:01:33.762440 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:33.762446 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:33.762505 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:33.762586 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:33.763219 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.763740 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:33.764413 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.764558 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:33.767257 (+ 2699us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:54529: BlockingRecv error: recv got EOF from 127.13.94.130:54529 (error 108)
Metrics: {"server-negotiator.queue_time_us":530}
Apr 30 08:01:33 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:33 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:33.937129 23548 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:33.931668 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:34141)
0430 08:01:33.932074 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:33.932088 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:33.932189 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:33.932440 (+ 251us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:33.932442 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:33.932611 (+ 169us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:33.932819 (+ 208us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:33.932825 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.933674 (+ 849us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:33.933676 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:33.934062 (+ 386us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:33.934067 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.934157 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:33.934706 (+ 549us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:33.934718 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:33.936920 (+ 2202us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":222,"threads_started":1}
W20260430 08:01:33.937208 23546 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:33.931949 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:34141 (local address 127.13.94.190:37101)
0430 08:01:33.932104 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:33.932107 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:33.932137 (+ 30us) server_negotiation.cc:408] Connection header received
0430 08:01:33.932275 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:33.932278 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:33.932329 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:33.932404 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:33.932971 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.933561 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:33.934191 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:33.934385 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:33.937070 (+ 2685us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:34141: BlockingRecv error: recv got EOF from 127.13.94.131:34141 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Apr 30 08:01:34 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:34 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:34.589740 23550 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:34.584558 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:33315)
0430 08:01:34.585008 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:34.585022 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:34.585139 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:34.585479 (+ 340us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:34.585482 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:34.585664 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:34.585873 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:34.585917 (+ 44us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.586651 (+ 734us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:34.586654 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:34.587064 (+ 410us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:34.587070 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.587172 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:34.587745 (+ 573us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:34.587759 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:34.589602 (+ 1843us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":160,"threads_started":1}
W20260430 08:01:34.589874 23549 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:34.584588 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:33315 (local address 127.13.94.190:37101)
0430 08:01:34.584940 (+ 352us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:34.584945 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:34.585099 (+ 154us) server_negotiation.cc:408] Connection header received
0430 08:01:34.585270 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:34.585273 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:34.585323 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:34.585407 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:34.586025 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.586541 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:34.587165 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.587293 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:34.589725 (+ 2432us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:33315: BlockingRecv error: recv got EOF from 127.13.94.129:33315 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":148,"threads_started":1}
Apr 30 08:01:34 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:34 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:34.773322 23551 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:34.768192 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:35919)
0430 08:01:34.768459 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:34.768473 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:34.768579 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:34.768953 (+ 374us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:34.768956 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:34.769129 (+ 173us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:34.769323 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:34.769329 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.770075 (+ 746us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:34.770078 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:34.770516 (+ 438us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:34.770522 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.770617 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:34.771157 (+ 540us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:34.771171 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:34.773199 (+ 2028us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
W20260430 08:01:34.773450 23549 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:34.768254 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:35919 (local address 127.13.94.190:37101)
0430 08:01:34.768402 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:34.768406 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:34.768540 (+ 134us) server_negotiation.cc:408] Connection header received
0430 08:01:34.768670 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:34.768673 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:34.768797 (+ 124us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:34.768881 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:34.769434 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.769924 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:34.770619 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.770754 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:34.773330 (+ 2576us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:35919: BlockingRecv error: recv got EOF from 127.13.94.130:35919 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Apr 30 08:01:34 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:34 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:34.943413 23552 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:34.938129 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:34869)
0430 08:01:34.938503 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:34.938516 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:34.938615 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:34.939081 (+ 466us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:34.939084 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:34.939290 (+ 206us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:34.939567 (+ 277us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:34.939574 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.940512 (+ 938us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:34.940516 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:34.940921 (+ 405us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:34.940927 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.941027 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:34.941571 (+ 544us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:34.941586 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:34.943299 (+ 1713us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":102,"threads_started":1}
W20260430 08:01:34.943585 23549 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:34.938299 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:34869 (local address 127.13.94.190:37101)
0430 08:01:34.938459 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:34.938465 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:34.938596 (+ 131us) server_negotiation.cc:408] Connection header received
0430 08:01:34.938745 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:34.938748 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:34.938804 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:34.939014 (+ 210us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:34.939703 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.940398 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:34.941031 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:34.941181 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:34.943428 (+ 2247us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:34869: BlockingRecv error: recv got EOF from 127.13.94.131:34869 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Apr 30 08:01:35 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:35 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:35.596019 23554 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:35.590537 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:58315)
0430 08:01:35.590909 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:35.590924 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:35.591129 (+ 205us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:35.591433 (+ 304us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:35.591437 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:35.591647 (+ 210us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:35.591843 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:35.591849 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.592622 (+ 773us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:35.592628 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:35.593140 (+ 512us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:35.593147 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.593302 (+ 155us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:35.593868 (+ 566us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:35.593881 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:35.595838 (+ 1957us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
W20260430 08:01:35.596098 23553 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:35.590560 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:58315 (local address 127.13.94.190:37101)
0430 08:01:35.590910 (+ 350us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:35.590916 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:35.591054 (+ 138us) server_negotiation.cc:408] Connection header received
0430 08:01:35.591253 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:35.591257 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:35.591313 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:35.591413 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:35.591966 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.592503 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:35.593286 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.593508 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:35.595976 (+ 2468us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:58315: BlockingRecv error: recv got EOF from 127.13.94.129:58315 (error 108)
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":111,"threads_started":1}
Apr 30 08:01:35 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:35 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:35.779323 23555 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:35.774280 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:33319)
0430 08:01:35.774637 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:35.774658 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:35.774806 (+ 148us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:35.775100 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:35.775103 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:35.775311 (+ 208us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:35.775512 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:35.775518 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.776275 (+ 757us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:35.776280 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:35.776657 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:35.776663 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.776824 (+ 161us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:35.777381 (+ 557us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:35.777395 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:35.779199 (+ 1804us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":91,"threads_started":1}
W20260430 08:01:35.779579 23553 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:35.774405 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:33319 (local address 127.13.94.190:37101)
0430 08:01:35.774608 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:35.774613 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:35.774738 (+ 125us) server_negotiation.cc:408] Connection header received
0430 08:01:35.774906 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:35.774910 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:35.774975 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:35.775074 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:35.775638 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.776148 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:35.776824 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.777018 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:35.779330 (+ 2312us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:33319: BlockingRecv error: recv got EOF from 127.13.94.130:33319 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Apr 30 08:01:35 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:35 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:35.949687 23556 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:35.944246 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:40835)
0430 08:01:35.944535 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:35.944549 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:35.944651 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:35.944970 (+ 319us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:35.944973 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:35.945184 (+ 211us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:35.945384 (+ 200us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:35.945390 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.946138 (+ 748us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:35.946144 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:35.946528 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:35.946534 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.946664 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:35.947310 (+ 646us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:35.947329 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:35.949559 (+ 2230us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":117,"threads_started":1}
W20260430 08:01:35.949836 23553 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:35.944345 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:40835 (local address 127.13.94.190:37101)
0430 08:01:35.944481 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:35.944485 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:35.944618 (+ 133us) server_negotiation.cc:408] Connection header received
0430 08:01:35.944779 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:35.944781 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:35.944840 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:35.944946 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:35.945529 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.946013 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:35.946648 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:35.946839 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:35.949685 (+ 2846us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:40835: BlockingRecv error: recv got EOF from 127.13.94.131:40835 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Apr 30 08:01:36 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:36 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:36.602005 23557 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:36.596994 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:56921)
0430 08:01:36.597360 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:36.597373 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:36.597463 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:36.597693 (+ 230us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:36.597695 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:36.597876 (+ 181us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:36.598064 (+ 188us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:36.598069 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.598808 (+ 739us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:36.598810 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:36.599205 (+ 395us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:36.599211 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.599300 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:36.599806 (+ 506us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:36.599819 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:36.601835 (+ 2016us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":125,"threads_started":1}
W20260430 08:01:36.602116 23558 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:36.597161 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:56921 (local address 127.13.94.190:37101)
0430 08:01:36.597456 (+ 295us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:36.597459 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:36.597474 (+ 15us) server_negotiation.cc:408] Connection header received
0430 08:01:36.597533 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:36.597536 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:36.597582 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:36.597661 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:36.598193 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.598718 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:36.599298 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.599437 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:36.601986 (+ 2549us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:56921: BlockingRecv error: recv got EOF from 127.13.94.129:56921 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
Apr 30 08:01:36 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:36 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:36.785832 23559 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:36.780227 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:33367)
0430 08:01:36.780537 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:36.780553 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:36.780647 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:36.780952 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:36.780955 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:36.781199 (+ 244us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:36.781457 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:36.781466 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.782450 (+ 984us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:36.782454 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:36.782970 (+ 516us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:36.782980 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.783162 (+ 182us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:36.783751 (+ 589us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:36.783767 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:36.785657 (+ 1890us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
W20260430 08:01:36.785871 23558 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:36.780419 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:33367 (local address 127.13.94.190:37101)
0430 08:01:36.780591 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:36.780594 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:36.780609 (+ 15us) server_negotiation.cc:408] Connection header received
0430 08:01:36.780787 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:36.780790 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:36.780839 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:36.780913 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:36.781570 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.782285 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:36.783160 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.783301 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:36.785774 (+ 2473us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:33367: BlockingRecv error: recv got EOF from 127.13.94.130:33367 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Apr 30 08:01:36 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:36 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:36.955627 23560 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:36.950524 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:36283)
0430 08:01:36.950778 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:36.950791 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:36.950886 (+ 95us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:36.951108 (+ 222us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:36.951110 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:36.951288 (+ 178us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:36.951592 (+ 304us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:36.951600 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.952359 (+ 759us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:36.952362 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:36.952793 (+ 431us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:36.952800 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.952889 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:36.953450 (+ 561us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:36.953464 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:36.955451 (+ 1987us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
W20260430 08:01:36.955689 23558 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:36.950574 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:36283 (local address 127.13.94.190:37101)
0430 08:01:36.950711 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:36.950714 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:36.950855 (+ 141us) server_negotiation.cc:408] Connection header received
0430 08:01:36.950961 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:36.950964 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:36.951009 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:36.951077 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:36.951739 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.952249 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:36.952928 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:36.953105 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:36.955578 (+ 2473us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:36283: BlockingRecv error: recv got EOF from 127.13.94.131:36283 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Apr 30 08:01:37 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:37 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:37.608563 23561 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:37.603127 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:39995)
0430 08:01:37.603495 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:37.603510 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:37.603611 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:37.603875 (+ 264us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:37.603877 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:37.604060 (+ 183us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:37.604255 (+ 195us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:37.604260 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.605216 (+ 956us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:37.605224 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:37.605754 (+ 530us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:37.605764 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.605900 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:37.606526 (+ 626us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:37.606541 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:37.608442 (+ 1901us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
W20260430 08:01:37.608837 23562 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:37.603307 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:39995 (local address 127.13.94.190:37101)
0430 08:01:37.603568 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:37.603572 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:37.603587 (+ 15us) server_negotiation.cc:408] Connection header received
0430 08:01:37.603710 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:37.603713 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:37.603761 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:37.603845 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:37.604392 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.605091 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:37.605911 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.606195 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:37.608640 (+ 2445us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:39995: BlockingRecv error: recv got EOF from 127.13.94.129:39995 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
Apr 30 08:01:37 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:37 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:37.791778 23563 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:37.786686 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:40211)
0430 08:01:37.786989 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:37.787002 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:37.787103 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:37.787339 (+ 236us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:37.787341 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:37.787521 (+ 180us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:37.787708 (+ 187us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:37.787714 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.788471 (+ 757us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:37.788473 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:37.788899 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:37.788906 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.788996 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:37.789547 (+ 551us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:37.789585 (+ 38us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:37.791655 (+ 2070us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
W20260430 08:01:37.791957 23562 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:37.786754 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:40211 (local address 127.13.94.190:37101)
0430 08:01:37.786914 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:37.786918 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:37.787071 (+ 153us) server_negotiation.cc:408] Connection header received
0430 08:01:37.787181 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:37.787183 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:37.787235 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:37.787304 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:37.787822 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.788353 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:37.789066 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.789230 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:37.791817 (+ 2587us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:40211: BlockingRecv error: recv got EOF from 127.13.94.130:40211 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Apr 30 08:01:37 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:37 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:37.961378 23564 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:37.956511 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:58385)
0430 08:01:37.956869 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:37.956882 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:37.956984 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:37.957229 (+ 245us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:37.957231 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:37.957409 (+ 178us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:37.957605 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:37.957611 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.958330 (+ 719us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:37.958333 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:37.958698 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:37.958703 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.958792 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:37.959330 (+ 538us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:37.959344 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:37.961260 (+ 1916us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
W20260430 08:01:37.961577 23562 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:37.956591 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:58385 (local address 127.13.94.190:37101)
0430 08:01:37.956793 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:37.956799 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:37.956953 (+ 154us) server_negotiation.cc:408] Connection header received
0430 08:01:37.957061 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:37.957065 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:37.957115 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:37.957197 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:37.957719 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.958232 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:37.958814 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:37.958968 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:37.961386 (+ 2418us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:58385: BlockingRecv error: recv got EOF from 127.13.94.131:58385 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Apr 30 08:01:38 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:38 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:38.615010 23565 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:38.609657 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:37011)
0430 08:01:38.609964 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:38.609978 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:38.610080 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:38.610379 (+ 299us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:38.610382 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:38.610567 (+ 185us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:38.610760 (+ 193us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:38.610765 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.611600 (+ 835us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:38.611604 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:38.611998 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:38.612005 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.612122 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:38.612812 (+ 690us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:38.612826 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:38.614893 (+ 2067us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
W20260430 08:01:38.615254 23566 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:38.609760 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:37011 (local address 127.13.94.190:37101)
0430 08:01:38.610018 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:38.610021 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:38.610037 (+ 16us) server_negotiation.cc:408] Connection header received
0430 08:01:38.610178 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:38.610181 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:38.610239 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:38.610333 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:38.610872 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.611455 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:38.612135 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.612293 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:38.615040 (+ 2747us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:37011: BlockingRecv error: recv got EOF from 127.13.94.129:37011 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
Apr 30 08:01:38 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:38 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:38.798506 23567 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:38.792898 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:37315)
0430 08:01:38.793278 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:38.793297 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:38.793486 (+ 189us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:38.793876 (+ 390us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:38.793879 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:38.794140 (+ 261us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:38.794360 (+ 220us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:38.794367 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.795394 (+ 1027us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:38.795398 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:38.795798 (+ 400us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:38.795804 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.795905 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:38.796438 (+ 533us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:38.796451 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:38.798371 (+ 1920us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":93,"threads_started":1}
W20260430 08:01:38.798666 23566 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:38.792984 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:37315 (local address 127.13.94.190:37101)
0430 08:01:38.793152 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:38.793157 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:38.793461 (+ 304us) server_negotiation.cc:408] Connection header received
0430 08:01:38.793664 (+ 203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:38.793671 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:38.793744 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:38.793827 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:38.794477 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.795277 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:38.795914 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.796059 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:38.798527 (+ 2468us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:37315: BlockingRecv error: recv got EOF from 127.13.94.130:37315 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Apr 30 08:01:38 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:38 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:38.967682 23568 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:38.962270 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:59631)
0430 08:01:38.962655 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:38.962670 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:38.962814 (+ 144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:38.963186 (+ 372us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:38.963193 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:38.963456 (+ 263us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:38.963670 (+ 214us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:38.963676 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.964455 (+ 779us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:38.964459 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:38.964877 (+ 418us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:38.964883 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.964973 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:38.965547 (+ 574us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:38.965561 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:38.967561 (+ 2000us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":93,"threads_started":1}
W20260430 08:01:38.967832 23566 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:38.962324 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:59631 (local address 127.13.94.190:37101)
0430 08:01:38.962485 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:38.962490 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:38.962759 (+ 269us) server_negotiation.cc:408] Connection header received
0430 08:01:38.962917 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:38.962921 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:38.963000 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:38.963086 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:38.963806 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.964335 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:38.965002 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:38.965206 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:38.967709 (+ 2503us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:59631: BlockingRecv error: recv got EOF from 127.13.94.131:59631 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Apr 30 08:01:39 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:39 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:39.621284 23569 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:39.615893 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:37461)
0430 08:01:39.616197 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:39.616211 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:39.616319 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:39.616576 (+ 257us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:39.616578 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:39.616798 (+ 220us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:39.616992 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:39.616998 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.617791 (+ 793us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:39.617794 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:39.618216 (+ 422us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:39.618223 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.618321 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:39.619034 (+ 713us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:39.619051 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:39.621166 (+ 2115us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":112,"threads_started":1}
W20260430 08:01:39.621543 23570 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:39.616082 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:37461 (local address 127.13.94.190:37101)
0430 08:01:39.616319 (+ 237us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:39.616322 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:39.616335 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:39.616408 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:39.616411 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:39.616456 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:39.616527 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:39.617103 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.617675 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:39.618377 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.618522 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:39.621419 (+ 2897us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:37461: BlockingRecv error: recv got EOF from 127.13.94.129:37461 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
Apr 30 08:01:39 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:39 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:39.805027 23571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:39.799366 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:58897)
0430 08:01:39.799721 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:39.799739 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:39.799868 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:39.800178 (+ 310us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:39.800184 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:39.800418 (+ 234us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:39.800634 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:39.800640 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.801894 (+ 1254us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:39.801898 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:39.802296 (+ 398us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:39.802305 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.802433 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:39.802970 (+ 537us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:39.802982 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:39.804852 (+ 1870us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":98,"threads_started":1}
W20260430 08:01:39.805068 23570 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:39.799385 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:58897 (local address 127.13.94.190:37101)
0430 08:01:39.799532 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:39.799536 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:39.799822 (+ 286us) server_negotiation.cc:408] Connection header received
0430 08:01:39.799975 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:39.799979 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:39.800038 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:39.800134 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:39.800831 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.801749 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:39.802427 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.802647 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:39.804957 (+ 2310us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:58897: BlockingRecv error: recv got EOF from 127.13.94.130:58897 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Apr 30 08:01:39 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:39 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:39.973816 23572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:39.968637 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:56369)
0430 08:01:39.968994 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:39.969007 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:39.969113 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:39.969407 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:39.969410 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:39.969614 (+ 204us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:39.969812 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:39.969817 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.970581 (+ 764us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:39.970585 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:39.970979 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:39.970985 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.971079 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:39.971638 (+ 559us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:39.971652 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:39.973681 (+ 2029us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
W20260430 08:01:39.973978 23570 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:39.968787 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:56369 (local address 127.13.94.190:37101)
0430 08:01:39.968931 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:39.968935 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:39.969070 (+ 135us) server_negotiation.cc:408] Connection header received
0430 08:01:39.969192 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:39.969195 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:39.969242 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:39.969320 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:39.969934 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.970462 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:39.971077 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:39.971206 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:39.973826 (+ 2620us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:56369: BlockingRecv error: recv got EOF from 127.13.94.131:56369 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Apr 30 08:01:40 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:40 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:40.630535 23573 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:40.622161 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:37331)
0430 08:01:40.622434 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:40.622447 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:40.622539 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:40.622805 (+ 266us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:40.622808 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:40.623004 (+ 196us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:40.623197 (+ 193us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:40.623203 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.623928 (+ 725us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:40.623931 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:40.624308 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:40.624313 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.624456 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:40.628074 (+ 3618us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:40.628090 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:40.630334 (+ 2244us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
W20260430 08:01:40.630725 23574 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:40.622215 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:37331 (local address 127.13.94.190:37101)
0430 08:01:40.622475 (+ 260us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:40.622478 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:40.622492 (+ 14us) server_negotiation.cc:408] Connection header received
0430 08:01:40.622643 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:40.622646 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:40.622694 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:40.622775 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:40.623294 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.623834 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:40.624448 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.624598 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:40.630593 (+ 5995us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:37331: BlockingRecv error: recv got EOF from 127.13.94.129:37331 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":64,"threads_started":1}
Apr 30 08:01:40 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:40 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:40.811275 23575 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:40.806018 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:43471)
0430 08:01:40.806277 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:40.806290 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:40.806394 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:40.806680 (+ 286us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:40.806683 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:40.806862 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:40.807065 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:40.807070 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.807853 (+ 783us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:40.807856 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:40.808232 (+ 376us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:40.808237 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.808331 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:40.808959 (+ 628us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:40.808973 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:40.811102 (+ 2129us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
W20260430 08:01:40.811339 23574 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:40.806089 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:43471 (local address 127.13.94.190:37101)
0430 08:01:40.806240 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:40.806244 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:40.806360 (+ 116us) server_negotiation.cc:408] Connection header received
0430 08:01:40.806513 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:40.806516 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:40.806565 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:40.806645 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:40.807184 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.807744 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:40.808333 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.808507 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:40.811234 (+ 2727us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:43471: BlockingRecv error: recv got EOF from 127.13.94.130:43471 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 30 08:01:40 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:40 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:40.979753 23576 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:40.974581 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:47623)
0430 08:01:40.974925 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:40.974943 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:40.975062 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:40.975348 (+ 286us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:40.975353 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:40.975591 (+ 238us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:40.975864 (+ 273us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:40.975871 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.976601 (+ 730us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:40.976605 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:40.977059 (+ 454us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:40.977066 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.977163 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:40.977733 (+ 570us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:40.977747 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:40.979626 (+ 1879us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":134,"threads_started":1}
W20260430 08:01:40.979892 23574 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:40.974683 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:47623 (local address 127.13.94.190:37101)
0430 08:01:40.974837 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:40.974841 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:40.975030 (+ 189us) server_negotiation.cc:408] Connection header received
0430 08:01:40.975153 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:40.975156 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:40.975215 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:40.975296 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:40.975984 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.976491 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:40.977158 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:40.977281 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:40.979777 (+ 2496us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:47623: BlockingRecv error: recv got EOF from 127.13.94.131:47623 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 30 08:01:41 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:41 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:41.636209 23577 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:41.631311 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:56581)
0430 08:01:41.631586 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:41.631600 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:41.631680 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:41.631969 (+ 289us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:41.631972 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:41.632149 (+ 177us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:41.632339 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:41.632344 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.633276 (+ 932us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:41.633279 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:41.633767 (+ 488us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:41.633775 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.633871 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:41.634360 (+ 489us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:41.634372 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:41.636093 (+ 1721us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
W20260430 08:01:41.636404 23578 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:41.631367 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:56581 (local address 127.13.94.190:37101)
0430 08:01:41.631631 (+ 264us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:41.631635 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:41.631647 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:01:41.631771 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:41.631774 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:41.631820 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:41.631928 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:41.632458 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.633179 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:41.633867 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.634002 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:41.636249 (+ 2247us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:56581: BlockingRecv error: recv got EOF from 127.13.94.129:56581 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":61,"threads_started":1}
Apr 30 08:01:41 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:41 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:41.817706 23579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:41.812272 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:34207)
0430 08:01:41.812538 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:41.812555 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:41.812664 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:41.813006 (+ 342us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:41.813009 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:41.813188 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:41.813378 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:41.813383 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.814226 (+ 843us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:41.814229 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:41.814695 (+ 466us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:41.814704 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.814856 (+ 152us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:41.815564 (+ 708us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:41.815577 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:41.817530 (+ 1953us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
W20260430 08:01:41.817750 23578 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:41.812368 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:34207 (local address 127.13.94.190:37101)
0430 08:01:41.812530 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:41.812534 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:41.812634 (+ 100us) server_negotiation.cc:408] Connection header received
0430 08:01:41.812837 (+ 203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:41.812841 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:41.812892 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:41.812972 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:41.813520 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.814075 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:41.814841 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.815063 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:41.817650 (+ 2587us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:34207: BlockingRecv error: recv got EOF from 127.13.94.130:34207 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Apr 30 08:01:41 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:41 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:41.985932 23580 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:41.980665 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:57543)
0430 08:01:41.981116 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:41.981132 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:41.981252 (+ 120us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:41.981557 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:41.981560 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:41.981754 (+ 194us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:41.981948 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:41.981954 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.982684 (+ 730us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:41.982688 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:41.983087 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:41.983094 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.983198 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:41.983761 (+ 563us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:41.983775 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:41.985756 (+ 1981us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":112,"threads_started":1}
W20260430 08:01:41.986029 23578 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:41.980916 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:57543 (local address 127.13.94.190:37101)
0430 08:01:41.981070 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:41.981074 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:41.981215 (+ 141us) server_negotiation.cc:408] Connection header received
0430 08:01:41.981340 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:41.981344 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:41.981404 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:41.981498 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:41.982059 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.982553 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:41.983199 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:41.983333 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:41.985908 (+ 2575us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:57543: BlockingRecv error: recv got EOF from 127.13.94.131:57543 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Apr 30 08:01:42 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:42 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:42.642726 23581 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:42.637230 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:59701)
0430 08:01:42.637676 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:42.637693 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:42.637809 (+ 116us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:42.638136 (+ 327us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:42.638139 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:42.638318 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:42.638510 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:42.638515 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.639323 (+ 808us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:42.639327 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:42.639763 (+ 436us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:42.639771 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.639873 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:42.640420 (+ 547us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:42.640434 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:42.642607 (+ 2173us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":81,"threads_started":1}
W20260430 08:01:42.642957 23582 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:42.637344 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:59701 (local address 127.13.94.190:37101)
0430 08:01:42.637866 (+ 522us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:42.637871 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:42.637887 (+ 16us) server_negotiation.cc:408] Connection header received
0430 08:01:42.637928 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:42.637931 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:42.638002 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:42.638099 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:42.638672 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.639211 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:42.639886 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.640031 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:42.642772 (+ 2741us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:59701: BlockingRecv error: recv got EOF from 127.13.94.129:59701 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
Apr 30 08:01:42 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:42 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:42.823390 23583 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:42.818607 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:33809)
0430 08:01:42.818903 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:42.818916 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:42.819019 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:42.819279 (+ 260us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:42.819282 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:42.819464 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:42.819675 (+ 211us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:42.819685 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.820451 (+ 766us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:42.820455 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:42.820890 (+ 435us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:42.820898 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.820993 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:42.821513 (+ 520us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:42.821526 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:42.823270 (+ 1744us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
W20260430 08:01:42.823592 23582 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:42.818664 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:33809 (local address 127.13.94.190:37101)
0430 08:01:42.818799 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:42.818818 (+ 19us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:42.819007 (+ 189us) server_negotiation.cc:408] Connection header received
0430 08:01:42.819111 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:42.819114 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:42.819169 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:42.819240 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:42.819824 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.820335 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:42.820989 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.821148 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:42.823446 (+ 2298us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:33809: BlockingRecv error: recv got EOF from 127.13.94.130:33809 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Apr 30 08:01:42 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:42 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:42.991777 23584 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:42.986959 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:34897)
0430 08:01:42.987231 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:42.987244 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:42.987351 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:42.987635 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:42.987638 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:42.987820 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:42.988023 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:42.988028 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.988830 (+ 802us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:42.988833 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:42.989209 (+ 376us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:42.989215 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.989311 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:42.989858 (+ 547us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:42.989871 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:42.991656 (+ 1785us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
W20260430 08:01:42.992105 23582 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:42.986923 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:34897 (local address 127.13.94.190:37101)
0430 08:01:42.987080 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:42.987083 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:42.987311 (+ 228us) server_negotiation.cc:408] Connection header received
0430 08:01:42.987432 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:42.987434 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:42.987483 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:42.987554 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:42.988126 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.988671 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:42.989309 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:42.989450 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:42.991955 (+ 2505us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:34897: BlockingRecv error: recv got EOF from 127.13.94.131:34897 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Apr 30 08:01:43 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:43 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:43.649367 23585 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:43.643753 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:58943)
0430 08:01:43.644052 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:43.644067 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:43.644165 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:43.644506 (+ 341us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:43.644509 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:43.644755 (+ 246us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:43.644984 (+ 229us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:43.644991 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.645878 (+ 887us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:43.645882 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:43.646283 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:43.646290 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.646390 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:43.646995 (+ 605us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:43.647009 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:43.649184 (+ 2175us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
W20260430 08:01:43.649495 23586 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:43.643840 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:58943 (local address 127.13.94.190:37101)
0430 08:01:43.644164 (+ 324us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:43.644168 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:43.644185 (+ 17us) server_negotiation.cc:408] Connection header received
0430 08:01:43.644234 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:43.644238 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:43.644290 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:43.644390 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:43.645130 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.645742 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:43.646413 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.646613 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:43.649337 (+ 2724us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:58943: BlockingRecv error: recv got EOF from 127.13.94.129:58943 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
Apr 30 08:01:43 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:43 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:43.830188 23587 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:43.824364 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:41313)
0430 08:01:43.824767 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:43.824788 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:43.824911 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:43.825224 (+ 313us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:43.825227 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:43.825415 (+ 188us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:43.825800 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:43.825809 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.826658 (+ 849us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:43.826665 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:43.827215 (+ 550us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:43.827225 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.827347 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:43.827950 (+ 603us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:43.827964 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:43.830063 (+ 2099us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
W20260430 08:01:43.830359 23586 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:43.824501 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:41313 (local address 127.13.94.190:37101)
0430 08:01:43.824646 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:43.824649 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:43.824893 (+ 244us) server_negotiation.cc:408] Connection header received
0430 08:01:43.825007 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:43.825010 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:43.825061 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:43.825147 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:43.825964 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.826542 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:43.827328 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.827475 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:43.830206 (+ 2731us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:41313: BlockingRecv error: recv got EOF from 127.13.94.130:41313 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Apr 30 08:01:43 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:43 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:43.997555 23588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:43.992609 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:49075)
0430 08:01:43.993006 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:43.993021 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:43.993136 (+ 115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:43.993406 (+ 270us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:43.993410 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:43.993595 (+ 185us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:43.993821 (+ 226us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:43.993827 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.994530 (+ 703us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:43.994533 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:43.994949 (+ 416us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:43.994955 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.995056 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:43.995603 (+ 547us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:43.995617 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:43.997441 (+ 1824us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":139,"threads_started":1}
W20260430 08:01:43.997726 23586 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:43.992633 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:49075 (local address 127.13.94.190:37101)
0430 08:01:43.992812 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:43.992816 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:43.993098 (+ 282us) server_negotiation.cc:408] Connection header received
0430 08:01:43.993228 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:43.993231 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:43.993280 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:43.993348 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:43.993927 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.994417 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:43.995262 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:43.995402 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:43.997571 (+ 2169us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:49075: BlockingRecv error: recv got EOF from 127.13.94.131:49075 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Apr 30 08:01:44 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:44 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:44.655460 23589 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:44.650235 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:48401)
0430 08:01:44.650538 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:44.650552 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:44.650646 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:44.651022 (+ 376us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:44.651025 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:44.651198 (+ 173us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:44.651394 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:44.651399 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:44.652181 (+ 782us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:44.652183 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:44.652549 (+ 366us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:44.652554 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:44.652638 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:44.653346 (+ 708us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:44.653360 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:44.655336 (+ 1976us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
W20260430 08:01:44.655694 23590 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:44.650368 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:48401 (local address 127.13.94.190:37101)
0430 08:01:44.650608 (+ 240us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:44.650611 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:44.650626 (+ 15us) server_negotiation.cc:408] Connection header received
0430 08:01:44.650844 (+ 218us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:44.650848 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:44.650901 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:44.650992 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:44.651510 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:44.652084 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:44.652662 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:44.653069 (+ 407us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:44.655471 (+ 2402us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:48401: BlockingRecv error: recv got EOF from 127.13.94.129:48401 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":59,"threads_started":1}
Apr 30 08:01:44 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:44 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:44.835819 23591 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:44.831013 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:38645)
0430 08:01:44.831294 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:44.831307 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:44.831419 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:44.831694 (+ 275us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:44.831697 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:44.831885 (+ 188us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:44.832084 (+ 199us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:44.832089 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:44.832848 (+ 759us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:44.832851 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:44.833241 (+ 390us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:44.833247 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:44.833349 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:44.833918 (+ 569us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:44.833934 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:44.835699 (+ 1765us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
W20260430 08:01:44.836028 23590 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:44.831176 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:38645 (local address 127.13.94.190:37101)
0430 08:01:44.831316 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:44.831320 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:44.831383 (+ 63us) server_negotiation.cc:408] Connection header received
0430 08:01:44.831506 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:44.831508 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:44.831556 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:44.831633 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:44.832192 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:44.832685 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:44.833343 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:44.833480 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:44.835888 (+ 2408us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:38645: BlockingRecv error: recv got EOF from 127.13.94.130:38645 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Apr 30 08:01:44 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:44 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:45.003208 23592 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:44.998344 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:54057)
0430 08:01:44.998623 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:44.998636 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:44.998737 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:44.999007 (+ 270us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:44.999010 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:44.999189 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:44.999380 (+ 191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:44.999385 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.000105 (+ 720us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:45.000108 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:45.000547 (+ 439us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:45.000554 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.000693 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:45.001300 (+ 607us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:45.001314 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:45.003085 (+ 1771us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
W20260430 08:01:45.003434 23590 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:44.998541 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:54057 (local address 127.13.94.190:37101)
0430 08:01:44.998670 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:44.998673 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:44.998686 (+ 13us) server_negotiation.cc:408] Connection header received
0430 08:01:44.998822 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:44.998825 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:44.998879 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:44.998978 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:44.999478 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:44.999997 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:45.000680 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.000972 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:45.003275 (+ 2303us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:54057: BlockingRecv error: recv got EOF from 127.13.94.131:54057 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Apr 30 08:01:45 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:45 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:45.665215 23593 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:45.656474 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:37757)
0430 08:01:45.656925 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:45.656945 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:45.657093 (+ 148us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:45.657374 (+ 281us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:45.657378 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:45.657623 (+ 245us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:45.657845 (+ 222us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:45.657852 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.658621 (+ 769us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:45.658627 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:45.659090 (+ 463us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:45.659097 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.659234 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:45.662978 (+ 3744us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:45.662996 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:45.665043 (+ 2047us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":116,"threads_started":1}
W20260430 08:01:45.665346 23594 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:45.656529 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:37757 (local address 127.13.94.190:37101)
0430 08:01:45.656926 (+ 397us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:45.656933 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:45.657039 (+ 106us) server_negotiation.cc:408] Connection header received
0430 08:01:45.657200 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:45.657208 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:45.657255 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:45.657353 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:45.657974 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.658501 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:45.659215 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.659425 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:45.665210 (+ 5785us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:37757: BlockingRecv error: recv got EOF from 127.13.94.129:37757 (error 108)
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
Apr 30 08:01:45 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:45 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:45.842489 23595 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:45.836935 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:47897)
0430 08:01:45.837277 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:45.837291 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:45.837443 (+ 152us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:45.837728 (+ 285us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:45.837775 (+ 47us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:45.838024 (+ 249us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:45.838224 (+ 200us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:45.838230 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.839104 (+ 874us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:45.839108 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:45.839493 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:45.839499 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.839626 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:45.840197 (+ 571us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:45.840211 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:45.842313 (+ 2102us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
W20260430 08:01:45.842551 23594 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:45.837037 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:47897 (local address 127.13.94.190:37101)
0430 08:01:45.837212 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:45.837218 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:45.837389 (+ 171us) server_negotiation.cc:408] Connection header received
0430 08:01:45.837552 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:45.837556 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:45.837614 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:45.837702 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:45.838344 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.838963 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:45.839608 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:45.839880 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:45.842443 (+ 2563us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:47897: BlockingRecv error: recv got EOF from 127.13.94.130:47897 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Apr 30 08:01:46 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:46 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.131@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:46.009732 23596 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:46.004200 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.131:55979)
0430 08:01:46.004494 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:46.004509 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:46.004621 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:46.004976 (+ 355us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:46.004978 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:46.005168 (+ 190us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:46.005376 (+ 208us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:46.005382 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.006240 (+ 858us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:46.006243 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:46.006729 (+ 486us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:46.006740 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.006887 (+ 147us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:46.007652 (+ 765us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:46.007666 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:46.009612 (+ 1946us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
W20260430 08:01:46.009935 23594 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:46.004298 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:55979 (local address 127.13.94.190:37101)
0430 08:01:46.004466 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:46.004470 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:46.004587 (+ 117us) server_negotiation.cc:408] Connection header received
0430 08:01:46.004787 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:46.004790 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:46.004845 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:46.004933 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:46.005510 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.006115 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:46.006984 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.007195 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:46.009792 (+ 2597us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.131:55979: BlockingRecv error: recv got EOF from 127.13.94.131:55979 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Apr 30 08:01:46 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:46 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.129@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:46.671629 23597 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:46.666426 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.129:37417)
0430 08:01:46.666714 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:46.666728 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:46.666827 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:46.667101 (+ 274us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:46.667104 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:46.667287 (+ 183us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:46.667476 (+ 189us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:46.667481 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.668231 (+ 750us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:46.668234 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:46.668610 (+ 376us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:46.668615 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.668770 (+ 155us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:46.669409 (+ 639us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:46.669426 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:46.671514 (+ 2088us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":102,"threads_started":1}
W20260430 08:01:46.671792 23598 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:46.666509 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:37417 (local address 127.13.94.190:37101)
0430 08:01:46.666770 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:46.666773 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:46.666788 (+ 15us) server_negotiation.cc:408] Connection header received
0430 08:01:46.666925 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:46.666928 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:46.666975 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:46.667072 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:46.667586 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.668127 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:46.668731 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.668867 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:46.671655 (+ 2788us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.129:37417: BlockingRecv error: recv got EOF from 127.13.94.129:37417 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
Apr 30 08:01:46 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
Apr 30 08:01:46 dist-test-slave-9bxp krb5kdc[22530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.94.130@KRBTEST.COM for oryx/127.13.94.190@KRBTEST.COM, Server not found in Kerberos database
I20260430 08:01:46.849311 23599 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:46.843683 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:37101 (local address 127.13.94.130:53235)
0430 08:01:46.843967 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:46.843981 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:46.844078 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:46.844364 (+ 286us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:46.844367 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:46.844552 (+ 185us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:46.844811 (+ 259us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:46.844819 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.845648 (+ 829us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:46.845651 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:46.846041 (+ 390us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:46.846048 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.846153 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:46.846721 (+ 568us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:46.846735 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:46.849126 (+ 2391us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.94.190:37101: Server oryx/127.13.94.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
W20260430 08:01:46.849393 23598 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:46.843812 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:53235 (local address 127.13.94.190:37101)
0430 08:01:46.844005 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:46.844008 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:46.844031 (+ 23us) server_negotiation.cc:408] Connection header received
0430 08:01:46.844171 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:46.844175 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:46.844234 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:46.844329 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:46.845001 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.845522 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:46.846160 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:46.846301 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:46.849273 (+ 2972us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.94.130:53235: BlockingRecv error: recv got EOF from 127.13.94.130:53235 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260430 08:01:46.909337 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 23122
I20260430 08:01:46.915830 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 23274
I20260430 08:01:46.921672 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 23406
I20260430 08:01:46.927616 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 23052
2026-04-30T08:01:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18980 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-04-30T08:01:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:46Z Disabled control of system clock
I20260430 08:01:46.959044 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:40693
--webserver_interface=127.13.94.190
--webserver_port=0
--builtin_ntp_servers=127.13.94.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:40693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260430 08:01:47.066190 23603 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:47.066424 23603 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:47.066468 23603 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:47.069825 23603 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:47.069907 23603 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:47.069924 23603 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:47.069943 23603 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:47.069962 23603 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:47.073869 23603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:40693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:40693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:47.074956 23603 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:47.076087 23603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:47.081959 23608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:47.082036 23609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:47.082147 23611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:47.082289 23603 server_base.cc:1061] running on GCE node
I20260430 08:01:47.082830 23603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:47.083770 23603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:47.084975 23603 hybrid_clock.cc:648] HybridClock initialized: now 1777536107084946 us; error 38 us; skew 500 ppm
I20260430 08:01:47.086944 23603 webserver.cc:492] Webserver started at http://127.13.94.190:39497/ using document root <none> and password file <none>
I20260430 08:01:47.087538 23603 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:47.087625 23603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:47.087862 23603 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:47.089627 23603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "14068331227641ffbf19691db13144e8"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "ad38bacd5dd2b55b0101c3be4be34edb"
server_key_iv: "cd62d05a145f0bc1dbecb70169d48382"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.090149 23603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "14068331227641ffbf19691db13144e8"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "ad38bacd5dd2b55b0101c3be4be34edb"
server_key_iv: "cd62d05a145f0bc1dbecb70169d48382"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.093825 23603 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260430 08:01:47.096287 23617 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.097379 23603 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260430 08:01:47.097519 23603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "14068331227641ffbf19691db13144e8"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "ad38bacd5dd2b55b0101c3be4be34edb"
server_key_iv: "cd62d05a145f0bc1dbecb70169d48382"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.097632 23603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:47.106397 23603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:47.107059 23603 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:47.107255 23603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:47.114744 23669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:40693 every 8 connection(s)
I20260430 08:01:47.114742 23603 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:40693
I20260430 08:01:47.115820 23603 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:47.118953 23670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:47.124167 23670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8: Bootstrap starting.
I20260430 08:01:47.124478 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 23603
I20260430 08:01:47.124598 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:47.124898 13690 external_mini_cluster.cc:1468] Setting key 871290e777f89f712b2be99461c964f1
I20260430 08:01:47.126811 23670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:47.127529 23670 log.cc:826] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:47.129593 23670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8: No bootstrap required, opened a new log
I20260430 08:01:47.132601 23673 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:47.126125 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.13.94.190:40693)
0430 08:01:47.126644 (+ 519us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:47.126653 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:47.126684 (+ 31us) server_negotiation.cc:408] Connection header received
0430 08:01:47.127231 (+ 547us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:47.127248 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:47.127576 (+ 328us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:47.127896 (+ 320us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:47.128395 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.129196 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:47.130083 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.130287 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:47.130971 (+ 684us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:47.130993 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:47.131004 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:47.131384 (+ 380us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:47.131420 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:47.131433 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:47.131519 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:47.131707 (+ 188us) server_negotiation.cc:300] Negotiation successful
0430 08:01:47.132004 (+ 297us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":124,"threads_started":1}
I20260430 08:01:47.132609 23670 raft_consensus.cc:359] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14068331227641ffbf19691db13144e8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40693 } }
I20260430 08:01:47.132982 23670 raft_consensus.cc:385] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:47.133072 23670 raft_consensus.cc:740] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14068331227641ffbf19691db13144e8, State: Initialized, Role: FOLLOWER
I20260430 08:01:47.133536 23670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14068331227641ffbf19691db13144e8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40693 } }
I20260430 08:01:47.133692 23670 raft_consensus.cc:399] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:47.133777 23670 raft_consensus.cc:493] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:47.133863 23670 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:47.134939 23670 raft_consensus.cc:515] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14068331227641ffbf19691db13144e8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40693 } }
I20260430 08:01:47.135242 23670 leader_election.cc:304] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14068331227641ffbf19691db13144e8; no voters:
I20260430 08:01:47.135481 23670 leader_election.cc:290] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:47.135694 23675 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:47.135977 23675 raft_consensus.cc:697] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [term 1 LEADER]: Becoming Leader. State: Replica: 14068331227641ffbf19691db13144e8, State: Running, Role: LEADER
I20260430 08:01:47.136278 23675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14068331227641ffbf19691db13144e8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40693 } }
I20260430 08:01:47.136600 23670 sys_catalog.cc:565] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:47.138049 23677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14068331227641ffbf19691db13144e8. Latest consensus state: current_term: 1 leader_uuid: "14068331227641ffbf19691db13144e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14068331227641ffbf19691db13144e8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40693 } } }
I20260430 08:01:47.138145 23676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14068331227641ffbf19691db13144e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14068331227641ffbf19691db13144e8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40693 } } }
I20260430 08:01:47.138293 23677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:47.138351 23676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8 [sys.catalog]: This master's current role is: LEADER
W20260430 08:01:47.141747 23690 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260430 08:01:47.141825 23690 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260430 08:01:47.141909 23688 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:47.142704 23688 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:47.147936 23688 catalog_manager.cc:1357] Generated new cluster ID: e88d7aa223a84e7cae577433c88a1e2f
I20260430 08:01:47.148025 23688 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:47.161777 23688 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:47.162716 23688 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:47.172005 23688 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 14068331227641ffbf19691db13144e8: Generated new TSK 0
I20260430 08:01:47.172734 23688 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 08:01:47.177435 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40693
--builtin_ntp_servers=127.13.94.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260430 08:01:47.281002 23694 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:47.281229 23694 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:47.281308 23694 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:47.284905 23694 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:47.284972 23694 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:47.285082 23694 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:01:47.289373 23694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:47.290609 23694 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:47.291875 23694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:47.299034 23699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:47.299221 23700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:47.299495 23702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:47.299552 23694 server_base.cc:1061] running on GCE node
I20260430 08:01:47.300053 23694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:47.300748 23694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:47.301918 23694 hybrid_clock.cc:648] HybridClock initialized: now 1777536107301887 us; error 45 us; skew 500 ppm
I20260430 08:01:47.304416 23694 webserver.cc:492] Webserver started at http://127.13.94.129:37209/ using document root <none> and password file <none>
I20260430 08:01:47.305235 23694 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:47.305310 23694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:47.305526 23694 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:47.308012 23694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "12924d4e9b24449891aed22c60492c64"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "19f2526d5473ea11ed9e5a670ade04f5"
server_key_iv: "5dcfd51b9f6339f8a358ee2937930de4"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.308645 23694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "12924d4e9b24449891aed22c60492c64"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "19f2526d5473ea11ed9e5a670ade04f5"
server_key_iv: "5dcfd51b9f6339f8a358ee2937930de4"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.313154 23694 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260430 08:01:47.315680 23708 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.316896 23694 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:47.317144 23694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "12924d4e9b24449891aed22c60492c64"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "19f2526d5473ea11ed9e5a670ade04f5"
server_key_iv: "5dcfd51b9f6339f8a358ee2937930de4"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.317271 23694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:47.332170 23694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:47.332947 23694 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:47.333165 23694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:47.333770 23694 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:47.334806 23694 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:47.334884 23694 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.334959 23694 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:47.335002 23694 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.345577 23694 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:38925
I20260430 08:01:47.345602 23821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:38925 every 8 connection(s)
I20260430 08:01:47.346638 23694 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:01:47.353945 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 23694
I20260430 08:01:47.354180 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:01:47.354558 13690 external_mini_cluster.cc:1468] Setting key 33d878477e59c03bc7b4704d20f42edf
I20260430 08:01:47.355727 23673 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:47.348645 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:46509 (local address 127.13.94.190:40693)
0430 08:01:47.348814 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:47.348817 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:47.349998 (+ 1181us) server_negotiation.cc:408] Connection header received
0430 08:01:47.350814 (+ 816us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:47.350817 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:47.350918 (+ 101us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:47.351062 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:47.352107 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.352615 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:47.353455 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.353642 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:47.355180 (+ 1538us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:47.355205 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:47.355207 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:47.355221 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:47.355243 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:47.355251 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:47.355338 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:47.355523 (+ 185us) server_negotiation.cc:300] Negotiation successful
0430 08:01:47.355585 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260430 08:01:47.356567 23824 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:47.349026 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:40693 (local address 127.13.94.129:46509)
0430 08:01:47.349684 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:47.349731 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:47.350589 (+ 858us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:47.351266 (+ 677us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:47.351275 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:47.351344 (+ 69us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:47.351915 (+ 571us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:47.351927 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.352855 (+ 928us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:47.352859 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:47.353330 (+ 471us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:47.353339 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.353553 (+ 214us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:47.354886 (+ 1333us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:01:47.354888 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:47.354926 (+ 38us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:01:47.354996 (+ 70us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:47.355344 (+ 348us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:47.355354 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:47.355360 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 08:01:47.355519 (+ 159us) client_negotiation.cc:241] Negotiation successful
0430 08:01:47.355690 (+ 171us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":99,"threads_started":1}
I20260430 08:01:47.356868 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40693
--builtin_ntp_servers=127.13.94.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260430 08:01:47.357869 23822 heartbeater.cc:344] Connected to a master server at 127.13.94.190:40693
I20260430 08:01:47.358157 23822 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:47.358846 23822 heartbeater.cc:507] Master 127.13.94.190:40693 requested a full tablet report, sending...
I20260430 08:01:47.360500 23634 ts_manager.cc:194] Registered new tserver with Master: 12924d4e9b24449891aed22c60492c64 (127.13.94.129:38925)
I20260430 08:01:47.361730 23634 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.94.129:46509
W20260430 08:01:47.467780 23825 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:47.468029 23825 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:47.468077 23825 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:47.471635 23825 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:47.471710 23825 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:47.471833 23825 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:01:47.475881 23825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:47.477078 23825 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:47.478350 23825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:47.485527 23833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:47.485455 23830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:47.485680 23825 server_base.cc:1061] running on GCE node
W20260430 08:01:47.485455 23831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:47.486140 23825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:47.486685 23825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:47.487831 23825 hybrid_clock.cc:648] HybridClock initialized: now 1777536107487800 us; error 48 us; skew 500 ppm
I20260430 08:01:47.489724 23825 webserver.cc:492] Webserver started at http://127.13.94.130:33597/ using document root <none> and password file <none>
I20260430 08:01:47.490304 23825 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:47.490358 23825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:47.490628 23825 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:47.492370 23825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "db898fbce9b24f33a214ed59715c4146"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "27fa291fde902c1b799f33e781e83514"
server_key_iv: "e3674e23e2d94bbcb71f35d70d308f54"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.492899 23825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "db898fbce9b24f33a214ed59715c4146"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "27fa291fde902c1b799f33e781e83514"
server_key_iv: "e3674e23e2d94bbcb71f35d70d308f54"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.496345 23825 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260430 08:01:47.498845 23839 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.500140 23825 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:47.500308 23825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "db898fbce9b24f33a214ed59715c4146"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "27fa291fde902c1b799f33e781e83514"
server_key_iv: "e3674e23e2d94bbcb71f35d70d308f54"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.500437 23825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:47.523823 23825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:47.524565 23825 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:47.524840 23825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:47.525476 23825 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:47.526538 23825 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:47.526612 23825 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.526685 23825 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:47.526737 23825 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.536067 23825 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:40025
I20260430 08:01:47.536087 23952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:40025 every 8 connection(s)
I20260430 08:01:47.537098 23825 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:01:47.544122 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 23825
I20260430 08:01:47.544274 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:01:47.544577 13690 external_mini_cluster.cc:1468] Setting key 0dd00335f4ba063153b519cdabc21f3e
I20260430 08:01:47.545255 23673 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:47.539102 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:55537 (local address 127.13.94.190:40693)
0430 08:01:47.539263 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:47.539266 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:47.540113 (+ 847us) server_negotiation.cc:408] Connection header received
0430 08:01:47.541020 (+ 907us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:47.541023 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:47.541077 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:47.541159 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:47.541988 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.542477 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:47.543119 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.543266 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:47.544558 (+ 1292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:47.544581 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:47.544586 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:47.544604 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:47.544630 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:47.544641 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:47.544773 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:47.545016 (+ 243us) server_negotiation.cc:300] Negotiation successful
0430 08:01:47.545095 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260430 08:01:47.545961 23955 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:47.539383 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:40693 (local address 127.13.94.130:55537)
0430 08:01:47.539981 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:47.540012 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:47.540826 (+ 814us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:47.541309 (+ 483us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:47.541317 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:47.541375 (+ 58us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:47.541861 (+ 486us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:47.541873 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.542581 (+ 708us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:47.542584 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:47.542978 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:47.542983 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.543175 (+ 192us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:47.544288 (+ 1113us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:01:47.544291 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:47.544325 (+ 34us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:01:47.544415 (+ 90us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:47.544821 (+ 406us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:47.544830 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:47.544837 (+ 7us) client_negotiation.cc:770] Sending connection context
0430 08:01:47.545007 (+ 170us) client_negotiation.cc:241] Negotiation successful
0430 08:01:47.545174 (+ 167us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":92,"threads_started":1}
I20260430 08:01:47.546774 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40693
--builtin_ntp_servers=127.13.94.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260430 08:01:47.547112 23953 heartbeater.cc:344] Connected to a master server at 127.13.94.190:40693
I20260430 08:01:47.547379 23953 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:47.548014 23953 heartbeater.cc:507] Master 127.13.94.190:40693 requested a full tablet report, sending...
I20260430 08:01:47.549335 23634 ts_manager.cc:194] Registered new tserver with Master: db898fbce9b24f33a214ed59715c4146 (127.13.94.130:40025)
I20260430 08:01:47.550168 23634 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.94.130:55537
W20260430 08:01:47.652956 23956 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:47.653198 23956 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:47.653245 23956 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:47.656629 23956 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:47.656685 23956 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:47.656847 23956 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:01:47.660694 23956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:47.661799 23956 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:47.662943 23956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:47.669566 23961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:47.669567 23962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:47.669567 23964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:47.670115 23956 server_base.cc:1061] running on GCE node
I20260430 08:01:47.670511 23956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:47.671157 23956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:47.672339 23956 hybrid_clock.cc:648] HybridClock initialized: now 1777536107672322 us; error 31 us; skew 500 ppm
I20260430 08:01:47.674232 23956 webserver.cc:492] Webserver started at http://127.13.94.131:41785/ using document root <none> and password file <none>
I20260430 08:01:47.674813 23956 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:47.674876 23956 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:47.675094 23956 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:47.676994 23956 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "ae9e4d0ba7bb428b9e1d9d63352ca994"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "a0c664e1f8d0e391a9a4dc59faad81dc"
server_key_iv: "9f558612d4958741733de0acc3e43269"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.677510 23956 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "ae9e4d0ba7bb428b9e1d9d63352ca994"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "a0c664e1f8d0e391a9a4dc59faad81dc"
server_key_iv: "9f558612d4958741733de0acc3e43269"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.681295 23956 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260430 08:01:47.684185 23970 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.685295 23956 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:47.685395 23956 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "ae9e4d0ba7bb428b9e1d9d63352ca994"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "a0c664e1f8d0e391a9a4dc59faad81dc"
server_key_iv: "9f558612d4958741733de0acc3e43269"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.685487 23956 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:47.702205 23956 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:47.702811 23956 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:47.702960 23956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:47.703557 23956 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:47.704499 23956 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:47.704546 23956 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.704586 23956 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:47.704600 23956 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.714713 23956 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:39241
I20260430 08:01:47.714737 24083 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:39241 every 8 connection(s)
I20260430 08:01:47.715719 23956 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:01:47.723740 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 23956
I20260430 08:01:47.723891 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:01:47.724227 13690 external_mini_cluster.cc:1468] Setting key 8aec4ecbd2fac9bb838ef673d087abf6
I20260430 08:01:47.724682 23673 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:47.717596 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:44885 (local address 127.13.94.190:40693)
0430 08:01:47.717779 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:47.717784 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:47.718645 (+ 861us) server_negotiation.cc:408] Connection header received
0430 08:01:47.719591 (+ 946us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:47.719594 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:47.719644 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:47.719721 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:47.720604 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.721289 (+ 685us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:47.722162 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.722364 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:47.723978 (+ 1614us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:47.724022 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:47.724025 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:47.724040 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:47.724061 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:47.724069 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:47.724158 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:47.724445 (+ 287us) server_negotiation.cc:300] Negotiation successful
0430 08:01:47.724498 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260430 08:01:47.725471 24086 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:47.717920 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:40693 (local address 127.13.94.131:44885)
0430 08:01:47.718509 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:47.718542 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:47.719397 (+ 855us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:47.719869 (+ 472us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:47.719876 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:47.719947 (+ 71us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:47.720430 (+ 483us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:47.720440 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.721441 (+ 1001us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:47.721445 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:47.721970 (+ 525us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:47.721979 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:47.722299 (+ 320us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:47.723609 (+ 1310us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:01:47.723613 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:47.723650 (+ 37us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:01:47.723721 (+ 71us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:47.724188 (+ 467us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:47.724198 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:47.724205 (+ 7us) client_negotiation.cc:770] Sending connection context
0430 08:01:47.724412 (+ 207us) client_negotiation.cc:241] Negotiation successful
0430 08:01:47.724593 (+ 181us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":101,"threads_started":1}
I20260430 08:01:47.726619 24084 heartbeater.cc:344] Connected to a master server at 127.13.94.190:40693
I20260430 08:01:47.726866 24084 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:47.727430 24084 heartbeater.cc:507] Master 127.13.94.190:40693 requested a full tablet report, sending...
I20260430 08:01:47.728461 23634 ts_manager.cc:194] Registered new tserver with Master: ae9e4d0ba7bb428b9e1d9d63352ca994 (127.13.94.131:39241)
I20260430 08:01:47.729130 23634 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.94.131:44885
I20260430 08:01:47.738651 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260430 08:01:47.741585 23673 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:47.740747 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.13.94.190:40693)
0430 08:01:47.740949 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:47.740953 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:47.740977 (+ 24us) server_negotiation.cc:408] Connection header received
0430 08:01:47.741012 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:47.741015 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:47.741061 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:47.741140 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:47.741439 (+ 299us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53668: BlockingRecv error: recv got EOF from 127.0.0.1:53668 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260430 08:01:47.741631 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 23694
I20260430 08:01:47.747963 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 23825
I20260430 08:01:47.753593 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 23956
I20260430 08:01:47.759313 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 23603
2026-04-30T08:01:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (830 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-04-30T08:01:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:47Z Disabled control of system clock
I20260430 08:01:47.788861 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:40181
--webserver_interface=127.13.94.190
--webserver_port=0
--builtin_ntp_servers=127.13.94.148:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:40181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260430 08:01:47.897954 24095 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:47.898181 24095 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:47.898226 24095 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:47.901546 24095 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:47.901619 24095 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:47.901636 24095 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:47.901655 24095 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:47.901672 24095 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:47.905916 24095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:40181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:40181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:47.907037 24095 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:47.908138 24095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:47.913734 24100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:47.913798 24101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:47.913798 24103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:47.914108 24095 server_base.cc:1061] running on GCE node
I20260430 08:01:47.914697 24095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:47.915539 24095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:47.916744 24095 hybrid_clock.cc:648] HybridClock initialized: now 1777536107916689 us; error 66 us; skew 500 ppm
I20260430 08:01:47.918789 24095 webserver.cc:492] Webserver started at http://127.13.94.190:36519/ using document root <none> and password file <none>
I20260430 08:01:47.919369 24095 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:47.919427 24095 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:47.919592 24095 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:47.921386 24095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "98c98f3d314047c685311e784ec1d2b8"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "b77f4b361b1681df1c914651641edc5a"
server_key_iv: "ca728b5d787f588101b327a8a0591aa1"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.921849 24095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "98c98f3d314047c685311e784ec1d2b8"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "b77f4b361b1681df1c914651641edc5a"
server_key_iv: "ca728b5d787f588101b327a8a0591aa1"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.925437 24095 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 08:01:47.927824 24109 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:47.929103 24095 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260430 08:01:47.929258 24095 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "98c98f3d314047c685311e784ec1d2b8"
format_stamp: "Formatted at 2026-04-30 08:01:47 on dist-test-slave-9bxp"
server_key: "b77f4b361b1681df1c914651641edc5a"
server_key_iv: "ca728b5d787f588101b327a8a0591aa1"
server_key_version: "encryptionkey@0"
I20260430 08:01:47.929382 24095 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:47.934522 24095 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:47.935204 24095 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:47.935410 24095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:47.943828 24161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:40181 every 8 connection(s)
I20260430 08:01:47.943840 24095 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:40181
I20260430 08:01:47.945106 24095 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:47.947983 24162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:47.953383 24162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8: Bootstrap starting.
I20260430 08:01:47.954365 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 24095
I20260430 08:01:47.954532 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:47.954792 13690 external_mini_cluster.cc:1468] Setting key 9d55611c313cabf536bb6c7b4e34f670
I20260430 08:01:47.956024 24162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:47.956928 24162 log.cc:826] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:47.959069 24162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8: No bootstrap required, opened a new log
I20260430 08:01:47.960141 24165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:47.955876 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.13.94.190:40181)
0430 08:01:47.956559 (+ 683us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:47.956570 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:47.956611 (+ 41us) server_negotiation.cc:408] Connection header received
0430 08:01:47.957400 (+ 789us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:47.957420 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:47.957662 (+ 242us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:47.958042 (+ 380us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:47.958864 (+ 822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:47.958870 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:47.958882 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:47.958893 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:47.958934 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:47.958948 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:47.959064 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:47.959201 (+ 137us) server_negotiation.cc:300] Negotiation successful
0430 08:01:47.959467 (+ 266us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":164,"threads_started":1}
I20260430 08:01:47.962181 24162 raft_consensus.cc:359] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c98f3d314047c685311e784ec1d2b8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40181 } }
I20260430 08:01:47.962373 24162 raft_consensus.cc:385] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:47.962411 24162 raft_consensus.cc:740] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c98f3d314047c685311e784ec1d2b8, State: Initialized, Role: FOLLOWER
I20260430 08:01:47.962785 24162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c98f3d314047c685311e784ec1d2b8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40181 } }
I20260430 08:01:47.962903 24162 raft_consensus.cc:399] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:47.962947 24162 raft_consensus.cc:493] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:47.963011 24162 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:47.963923 24162 raft_consensus.cc:515] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c98f3d314047c685311e784ec1d2b8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40181 } }
I20260430 08:01:47.964303 24162 leader_election.cc:304] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c98f3d314047c685311e784ec1d2b8; no voters:
I20260430 08:01:47.964600 24162 leader_election.cc:290] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:47.964907 24167 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:47.965137 24167 raft_consensus.cc:697] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [term 1 LEADER]: Becoming Leader. State: Replica: 98c98f3d314047c685311e784ec1d2b8, State: Running, Role: LEADER
I20260430 08:01:47.965551 24167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c98f3d314047c685311e784ec1d2b8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40181 } }
I20260430 08:01:47.965857 24162 sys_catalog.cc:565] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:47.967329 24168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98c98f3d314047c685311e784ec1d2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c98f3d314047c685311e784ec1d2b8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40181 } } }
I20260430 08:01:47.967389 24169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c98f3d314047c685311e784ec1d2b8. Latest consensus state: current_term: 1 leader_uuid: "98c98f3d314047c685311e784ec1d2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c98f3d314047c685311e784ec1d2b8" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 40181 } } }
I20260430 08:01:47.967450 24168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:47.967511 24169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8 [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:47.968077 24177 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:47.971197 24177 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:47.979476 24177 catalog_manager.cc:1357] Generated new cluster ID: 86e20a68e1254c2181ae1a3f6416b3a3
I20260430 08:01:47.979609 24177 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:47.986270 24177 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:47.986593 24177 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:47.995002 24177 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 98c98f3d314047c685311e784ec1d2b8: Generated new TSK 0
I20260430 08:01:47.995638 24177 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 08:01:48.004478 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40181
--builtin_ntp_servers=127.13.94.148:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260430 08:01:48.114249 24186 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:48.114470 24186 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:48.114522 24186 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:48.118162 24186 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:48.118234 24186 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:48.118346 24186 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:01:48.122716 24186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:48.123873 24186 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:48.125114 24186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:48.131664 24192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:48.131669 24194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:48.131728 24191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:48.132395 24186 server_base.cc:1061] running on GCE node
I20260430 08:01:48.132988 24186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:48.133579 24186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:48.134752 24186 hybrid_clock.cc:648] HybridClock initialized: now 1777536108134730 us; error 37 us; skew 500 ppm
I20260430 08:01:48.136650 24186 webserver.cc:492] Webserver started at http://127.13.94.129:44599/ using document root <none> and password file <none>
I20260430 08:01:48.137262 24186 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:48.137318 24186 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:48.137524 24186 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:48.139201 24186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "2f9abf788256487b936d5679a7749790"
format_stamp: "Formatted at 2026-04-30 08:01:48 on dist-test-slave-9bxp"
server_key: "904b97c5b175f9ce305ed350b008b60c"
server_key_iv: "c06f79bed2ad17bfc86c6d3683ded2e9"
server_key_version: "encryptionkey@0"
I20260430 08:01:48.139678 24186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "2f9abf788256487b936d5679a7749790"
format_stamp: "Formatted at 2026-04-30 08:01:48 on dist-test-slave-9bxp"
server_key: "904b97c5b175f9ce305ed350b008b60c"
server_key_iv: "c06f79bed2ad17bfc86c6d3683ded2e9"
server_key_version: "encryptionkey@0"
I20260430 08:01:48.143056 24186 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260430 08:01:48.145412 24200 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:48.146610 24186 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:48.146759 24186 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "2f9abf788256487b936d5679a7749790"
format_stamp: "Formatted at 2026-04-30 08:01:48 on dist-test-slave-9bxp"
server_key: "904b97c5b175f9ce305ed350b008b60c"
server_key_iv: "c06f79bed2ad17bfc86c6d3683ded2e9"
server_key_version: "encryptionkey@0"
I20260430 08:01:48.146883 24186 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:48.151934 24186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:48.152537 24186 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:48.152783 24186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:48.153339 24186 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:48.154318 24186 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:48.154392 24186 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:48.154453 24186 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:48.154511 24186 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:48.163631 24186 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:33089
I20260430 08:01:48.163657 24313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:33089 every 8 connection(s)
I20260430 08:01:48.164583 24186 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:01:48.170226 24165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:48.166483 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:37063 (local address 127.13.94.190:40181)
0430 08:01:48.166662 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:48.166666 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:48.167593 (+ 927us) server_negotiation.cc:408] Connection header received
0430 08:01:48.168459 (+ 866us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:48.168463 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:48.168516 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:48.168626 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:48.169703 (+ 1077us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:48.169707 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:48.169710 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:48.169722 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:48.169751 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:48.169759 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:48.169862 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:48.170036 (+ 174us) server_negotiation.cc:300] Negotiation successful
0430 08:01:48.170076 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260430 08:01:48.170588 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 24186
I20260430 08:01:48.170742 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:01:48.170964 24316 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:48.166780 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:40181 (local address 127.13.94.129:37063)
0430 08:01:48.167425 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:48.167459 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:48.168249 (+ 790us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:48.168819 (+ 570us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:48.168828 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:48.168929 (+ 101us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:48.169480 (+ 551us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:01:48.169482 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:48.169506 (+ 24us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:01:48.169554 (+ 48us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:48.169844 (+ 290us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:48.169851 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:48.169856 (+ 5us) client_negotiation.cc:770] Sending connection context
0430 08:01:48.170083 (+ 227us) client_negotiation.cc:241] Negotiation successful
0430 08:01:48.170191 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":102,"threads_started":1}
I20260430 08:01:48.171053 13690 external_mini_cluster.cc:1468] Setting key ba61bdef9b5fd3e41a74f97a9a229c26
I20260430 08:01:48.172469 24314 heartbeater.cc:344] Connected to a master server at 127.13.94.190:40181
I20260430 08:01:48.172798 24314 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:48.173271 24314 heartbeater.cc:507] Master 127.13.94.190:40181 requested a full tablet report, sending...
I20260430 08:01:48.173424 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40181
--builtin_ntp_servers=127.13.94.148:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260430 08:01:48.174765 24125 ts_manager.cc:194] Registered new tserver with Master: 2f9abf788256487b936d5679a7749790 (127.13.94.129:33089)
W20260430 08:01:48.279361 24317 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:48.279664 24317 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:48.279721 24317 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:48.283218 24317 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:48.283282 24317 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:48.283357 24317 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:01:48.287212 24317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:48.288300 24317 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:48.289449 24317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:48.296060 24323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:48.296121 24325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:48.296264 24322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:48.296553 24317 server_base.cc:1061] running on GCE node
I20260430 08:01:48.296998 24317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:48.297572 24317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:48.298810 24317 hybrid_clock.cc:648] HybridClock initialized: now 1777536108298783 us; error 36 us; skew 500 ppm
I20260430 08:01:48.301323 24317 webserver.cc:492] Webserver started at http://127.13.94.130:33321/ using document root <none> and password file <none>
I20260430 08:01:48.302245 24317 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:48.302350 24317 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:48.302592 24317 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:48.304388 24317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "5a8d7e0928c64c5caa463da80fa34d80"
format_stamp: "Formatted at 2026-04-30 08:01:48 on dist-test-slave-9bxp"
server_key: "0892d802d57aa15fe0676c0ae0da582e"
server_key_iv: "bd99864701586533d53744ec52412f81"
server_key_version: "encryptionkey@0"
I20260430 08:01:48.304975 24317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "5a8d7e0928c64c5caa463da80fa34d80"
format_stamp: "Formatted at 2026-04-30 08:01:48 on dist-test-slave-9bxp"
server_key: "0892d802d57aa15fe0676c0ae0da582e"
server_key_iv: "bd99864701586533d53744ec52412f81"
server_key_version: "encryptionkey@0"
I20260430 08:01:48.308570 24317 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260430 08:01:48.311206 24331 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:48.312388 24317 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260430 08:01:48.312552 24317 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "5a8d7e0928c64c5caa463da80fa34d80"
format_stamp: "Formatted at 2026-04-30 08:01:48 on dist-test-slave-9bxp"
server_key: "0892d802d57aa15fe0676c0ae0da582e"
server_key_iv: "bd99864701586533d53744ec52412f81"
server_key_version: "encryptionkey@0"
I20260430 08:01:48.312671 24317 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:48.317829 24317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:48.318606 24317 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:48.318826 24317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:48.319486 24317 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:48.320616 24317 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:48.320688 24317 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:48.320842 24317 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:48.320897 24317 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:48.331926 24317 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:40945
I20260430 08:01:48.331939 24444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:40945 every 8 connection(s)
I20260430 08:01:48.333070 24317 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:01:48.338930 24165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:48.335072 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:48987 (local address 127.13.94.190:40181)
0430 08:01:48.335224 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:48.335228 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:48.336166 (+ 938us) server_negotiation.cc:408] Connection header received
0430 08:01:48.337121 (+ 955us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:48.337125 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:48.337175 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:48.337263 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:48.338404 (+ 1141us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:48.338408 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:48.338411 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:48.338431 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:48.338451 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:48.338460 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:48.338575 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:48.338734 (+ 159us) server_negotiation.cc:300] Negotiation successful
0430 08:01:48.338776 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260430 08:01:48.339586 24447 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:48.335355 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:40181 (local address 127.13.94.130:48987)
0430 08:01:48.335977 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:48.336011 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:48.336909 (+ 898us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:48.337435 (+ 526us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:48.337444 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:48.337523 (+ 79us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:48.338148 (+ 625us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:01:48.338152 (+ 4us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:48.338180 (+ 28us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:01:48.338234 (+ 54us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:48.338559 (+ 325us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:48.338566 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:48.338572 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 08:01:48.338775 (+ 203us) client_negotiation.cc:241] Negotiation successful
0430 08:01:48.338933 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":107,"threads_started":1}
I20260430 08:01:48.339936 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 24317
I20260430 08:01:48.340066 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:01:48.340400 13690 external_mini_cluster.cc:1468] Setting key 22b8f228ff508b75ca4d4620caf07204
I20260430 08:01:48.341195 24445 heartbeater.cc:344] Connected to a master server at 127.13.94.190:40181
I20260430 08:01:48.341457 24445 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:48.341943 24445 heartbeater.cc:507] Master 127.13.94.190:40181 requested a full tablet report, sending...
I20260430 08:01:48.342585 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40181
--builtin_ntp_servers=127.13.94.148:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260430 08:01:48.343089 24125 ts_manager.cc:194] Registered new tserver with Master: 5a8d7e0928c64c5caa463da80fa34d80 (127.13.94.130:40945)
W20260430 08:01:48.448552 24448 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:48.448874 24448 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:48.449047 24448 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:48.452631 24448 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:48.452761 24448 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:48.452965 24448 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:01:48.457161 24448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:40181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:48.458365 24448 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:48.459616 24448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:48.467000 24453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:48.466950 24454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:48.466956 24456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:48.467178 24448 server_base.cc:1061] running on GCE node
I20260430 08:01:48.467662 24448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:48.468351 24448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:48.469564 24448 hybrid_clock.cc:648] HybridClock initialized: now 1777536108469549 us; error 36 us; skew 500 ppm
I20260430 08:01:48.471459 24448 webserver.cc:492] Webserver started at http://127.13.94.131:37103/ using document root <none> and password file <none>
I20260430 08:01:48.472050 24448 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:48.472136 24448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:48.472361 24448 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:48.474143 24448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "6cd05a0157574c389a0c3d3b8225061f"
format_stamp: "Formatted at 2026-04-30 08:01:48 on dist-test-slave-9bxp"
server_key: "5f1eb3231c22d218ddf421020f693b94"
server_key_iv: "dcc7bf059c543fc16ff4d82c385b6a53"
server_key_version: "encryptionkey@0"
I20260430 08:01:48.474658 24448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "6cd05a0157574c389a0c3d3b8225061f"
format_stamp: "Formatted at 2026-04-30 08:01:48 on dist-test-slave-9bxp"
server_key: "5f1eb3231c22d218ddf421020f693b94"
server_key_iv: "dcc7bf059c543fc16ff4d82c385b6a53"
server_key_version: "encryptionkey@0"
I20260430 08:01:48.478250 24448 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260430 08:01:48.480854 24462 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:48.482005 24448 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260430 08:01:48.482131 24448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "6cd05a0157574c389a0c3d3b8225061f"
format_stamp: "Formatted at 2026-04-30 08:01:48 on dist-test-slave-9bxp"
server_key: "5f1eb3231c22d218ddf421020f693b94"
server_key_iv: "dcc7bf059c543fc16ff4d82c385b6a53"
server_key_version: "encryptionkey@0"
I20260430 08:01:48.482246 24448 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:48.487062 24448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:48.487636 24448 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:48.487816 24448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:48.488368 24448 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:48.489406 24448 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:48.489452 24448 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:48.489519 24448 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:48.489552 24448 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:48.499123 24448 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:39657
I20260430 08:01:48.499145 24575 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:39657 every 8 connection(s)
I20260430 08:01:48.500159 24448 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:01:48.505844 24165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:48.502063 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:36409 (local address 127.13.94.190:40181)
0430 08:01:48.502229 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:48.502232 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:48.503151 (+ 919us) server_negotiation.cc:408] Connection header received
0430 08:01:48.503988 (+ 837us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:48.503991 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:48.504039 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:48.504150 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:48.505314 (+ 1164us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:48.505318 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:48.505321 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0430 08:01:48.505333 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:48.505357 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0430 08:01:48.505372 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:48.505472 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:48.505639 (+ 167us) server_negotiation.cc:300] Negotiation successful
0430 08:01:48.505680 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260430 08:01:48.506460 24578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:48.502336 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:40181 (local address 127.13.94.131:36409)
0430 08:01:48.502974 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:48.503008 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:48.503770 (+ 762us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:48.504312 (+ 542us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:48.504321 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:48.504382 (+ 61us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:48.505081 (+ 699us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0430 08:01:48.505083 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:48.505108 (+ 25us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0430 08:01:48.505162 (+ 54us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:48.505463 (+ 301us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:48.505470 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:48.505476 (+ 6us) client_negotiation.cc:770] Sending connection context
0430 08:01:48.505674 (+ 198us) client_negotiation.cc:241] Negotiation successful
0430 08:01:48.505801 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":99,"threads_started":1}
I20260430 08:01:48.507889 24576 heartbeater.cc:344] Connected to a master server at 127.13.94.190:40181
I20260430 08:01:48.508234 24576 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:48.508724 24576 heartbeater.cc:507] Master 127.13.94.190:40181 requested a full tablet report, sending...
I20260430 08:01:48.509177 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 24448
I20260430 08:01:48.509292 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:01:48.509598 13690 external_mini_cluster.cc:1468] Setting key 753499093608f832f7de0b28254311be
I20260430 08:01:48.510010 24125 ts_manager.cc:194] Registered new tserver with Master: 6cd05a0157574c389a0c3d3b8225061f (127.13.94.131:39657)
I20260430 08:01:48.511720 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260430 08:01:48.514549 24165 negotiation.cc:336] Failed RPC negotiation. Trace:
0430 08:01:48.513709 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.13.94.190:40181)
0430 08:01:48.513849 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:48.513852 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:48.514016 (+ 164us) server_negotiation.cc:408] Connection header received
0430 08:01:48.514114 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:48.514116 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:48.514166 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:48.514235 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:48.514417 (+ 182us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58730: BlockingRecv error: recv got EOF from 127.0.0.1:58730 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260430 08:01:48.515020 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 24186
I20260430 08:01:48.521399 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 24317
I20260430 08:01:48.527005 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 24448
I20260430 08:01:48.533022 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 24095
2026-04-30T08:01:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (773 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:01:48 dist-test-slave-9bxp krb5kdc[24586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:01:48 dist-test-slave-9bxp krb5kdc[24586](info): set up 2 sockets
Apr 30 08:01:48 dist-test-slave-9bxp krb5kdc[24586](info): commencing operation
krb5kdc: starting...
W20260430 08:01:50.537812 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.982s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:01:50 dist-test-slave-9bxp krb5kdc[24586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:01:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:50.684677 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:46737
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:46737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:50.786082 24602 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:50.786360 24602 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:50.786474 24602 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:50.789824 24602 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:01:50.789920 24602 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:50.789973 24602 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:50.790036 24602 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:01:50.790077 24602 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:01:50.794420 24602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:46737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:46737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:50.795563 24602 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:50.796468 24602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:50.802228 24607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:50.802225 24610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:50.802510 24608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:50.802497 24602 server_base.cc:1061] running on GCE node
I20260430 08:01:50.803001 24602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:50.803884 24602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:50.805032 24602 hybrid_clock.cc:648] HybridClock initialized: now 1777536110805021 us; error 49 us; skew 500 ppm
Apr 30 08:01:50 dist-test-slave-9bxp krb5kdc[24586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536110, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:50.808081 24602 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:01:50.809321 24602 webserver.cc:492] Webserver started at http://127.13.94.190:34305/ using document root <none> and password file <none>
I20260430 08:01:50.809906 24602 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:50.809955 24602 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:50.810173 24602 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:50.811874 24602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd"
format_stamp: "Formatted at 2026-04-30 08:01:50 on dist-test-slave-9bxp"
server_key: "e7288d2df1d51210bd8bc49933947b8e"
server_key_iv: "330dfeea05da3b399c847f8a010d1aed"
server_key_version: "encryptionkey@0"
I20260430 08:01:50.812389 24602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd"
format_stamp: "Formatted at 2026-04-30 08:01:50 on dist-test-slave-9bxp"
server_key: "e7288d2df1d51210bd8bc49933947b8e"
server_key_iv: "330dfeea05da3b399c847f8a010d1aed"
server_key_version: "encryptionkey@0"
I20260430 08:01:50.816363 24602 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260430 08:01:50.819149 24617 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:50.820298 24602 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:50.820451 24602 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd"
format_stamp: "Formatted at 2026-04-30 08:01:50 on dist-test-slave-9bxp"
server_key: "e7288d2df1d51210bd8bc49933947b8e"
server_key_iv: "330dfeea05da3b399c847f8a010d1aed"
server_key_version: "encryptionkey@0"
I20260430 08:01:50.820571 24602 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:50.853603 24602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:50.860307 24602 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:50.860565 24602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:50.868134 24602 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:46737
I20260430 08:01:50.868129 24669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:46737 every 8 connection(s)
I20260430 08:01:50.869297 24602 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:01:50.870947 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 24602
I20260430 08:01:50.871097 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:01:50.871408 13690 external_mini_cluster.cc:1468] Setting key cd02a707dbff383a97a1eeb319be51a4
I20260430 08:01:50.872476 24670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:50.878123 24670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd: Bootstrap starting.
Apr 30 08:01:50 dist-test-slave-9bxp krb5kdc[24586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:50.880278 24670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:50.881024 24670 log.cc:826] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:50.882984 24670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd: No bootstrap required, opened a new log
I20260430 08:01:50.885017 24673 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:50.872883 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.13.94.190:46737)
0430 08:01:50.873308 (+ 425us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:50.873318 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:50.873354 (+ 36us) server_negotiation.cc:408] Connection header received
0430 08:01:50.874050 (+ 696us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:50.874069 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:50.874355 (+ 286us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:50.874735 (+ 380us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:50.875746 (+ 1011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:50.876545 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:50.877330 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:50.877635 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:50.880056 (+ 2421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:50.880075 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:50.880086 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:50.880119 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:50.882340 (+ 2221us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:50.882815 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:50.882820 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:50.882825 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:50.882898 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:50.883179 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:50.883185 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:50.883187 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:50.883538 (+ 351us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:50.883755 (+ 217us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:50.884084 (+ 329us) server_negotiation.cc:300] Negotiation successful
0430 08:01:50.884339 (+ 255us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260430 08:01:50.885975 24670 raft_consensus.cc:359] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46737 } }
I20260430 08:01:50.886227 24670 raft_consensus.cc:385] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:50.886320 24670 raft_consensus.cc:740] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ccd203f9bc42afb1cae3f7a766b5cd, State: Initialized, Role: FOLLOWER
I20260430 08:01:50.886767 24670 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46737 } }
I20260430 08:01:50.886914 24670 raft_consensus.cc:399] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:01:50.886983 24670 raft_consensus.cc:493] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:01:50.887105 24670 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:50.888011 24670 raft_consensus.cc:515] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46737 } }
I20260430 08:01:50.888365 24670 leader_election.cc:304] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ccd203f9bc42afb1cae3f7a766b5cd; no voters:
I20260430 08:01:50.888664 24670 leader_election.cc:290] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:01:50.888844 24675 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:50.889086 24675 raft_consensus.cc:697] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [term 1 LEADER]: Becoming Leader. State: Replica: e7ccd203f9bc42afb1cae3f7a766b5cd, State: Running, Role: LEADER
I20260430 08:01:50.889528 24675 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46737 } }
I20260430 08:01:50.889884 24670 sys_catalog.cc:565] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:01:50.891665 24677 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ccd203f9bc42afb1cae3f7a766b5cd. Latest consensus state: current_term: 1 leader_uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46737 } } }
I20260430 08:01:50.891784 24677 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:50.891789 24676 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ccd203f9bc42afb1cae3f7a766b5cd" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 46737 } } }
I20260430 08:01:50.891918 24676 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd [sys.catalog]: This master's current role is: LEADER
I20260430 08:01:50.892360 24684 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:01:50.894917 24684 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:01:50.900458 24684 catalog_manager.cc:1357] Generated new cluster ID: 4c779c75274e4e32885e2765c233c381
I20260430 08:01:50.900553 24684 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:01:50.921105 24684 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:01:50.921931 24684 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:01:50.931547 24684 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd: Generated new TSK 0
I20260430 08:01:50.932317 24684 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:50.991799 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:46737
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:51.097872 24698 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:51.098119 24698 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:51.098222 24698 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:51.101763 24698 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:51.101838 24698 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:51.101980 24698 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:01:51.106357 24698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:51.107517 24698 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:51.108379 24698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260430 08:01:51.115367 24698 server_base.cc:1061] running on GCE node
W20260430 08:01:51.115319 24703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:51.115319 24704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:51.115309 24706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:51.115931 24698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:51.116478 24698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:51.117636 24698 hybrid_clock.cc:648] HybridClock initialized: now 1777536111117612 us; error 37 us; skew 500 ppm
Apr 30 08:01:51 dist-test-slave-9bxp krb5kdc[24586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536111, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:51.120416 24698 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:01:51.121520 24698 webserver.cc:492] Webserver started at http://127.13.94.129:43821/ using document root <none> and password file <none>
I20260430 08:01:51.122108 24698 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:51.122185 24698 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:51.122403 24698 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:51.124045 24698 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "11e8fa73361b4ab9ad1699235f74cc84"
format_stamp: "Formatted at 2026-04-30 08:01:51 on dist-test-slave-9bxp"
server_key: "9def902a30a01455b21e575ad46f5171"
server_key_iv: "3dd652320ee3bf3f3b3bf34cd4293834"
server_key_version: "encryptionkey@0"
I20260430 08:01:51.124601 24698 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "11e8fa73361b4ab9ad1699235f74cc84"
format_stamp: "Formatted at 2026-04-30 08:01:51 on dist-test-slave-9bxp"
server_key: "9def902a30a01455b21e575ad46f5171"
server_key_iv: "3dd652320ee3bf3f3b3bf34cd4293834"
server_key_version: "encryptionkey@0"
I20260430 08:01:51.128037 24698 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260430 08:01:51.130321 24713 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:51.131475 24698 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:01:51.131623 24698 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "11e8fa73361b4ab9ad1699235f74cc84"
format_stamp: "Formatted at 2026-04-30 08:01:51 on dist-test-slave-9bxp"
server_key: "9def902a30a01455b21e575ad46f5171"
server_key_iv: "3dd652320ee3bf3f3b3bf34cd4293834"
server_key_version: "encryptionkey@0"
I20260430 08:01:51.131737 24698 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:51.143308 24698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:51.145840 24698 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:51.146052 24698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:51.146646 24698 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:51.147595 24698 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:51.147661 24698 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:51.147732 24698 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:51.147774 24698 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:51.156934 24698 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:41611
I20260430 08:01:51.156970 24826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:41611 every 8 connection(s)
I20260430 08:01:51.157938 24698 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
Apr 30 08:01:51 dist-test-slave-9bxp krb5kdc[24586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536111, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:51.168011 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 24698
I20260430 08:01:51.168169 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:01:51.168494 13690 external_mini_cluster.cc:1468] Setting key b7c5ba001a8a3e7f98347d70fe457b5b
I20260430 08:01:51.170725 24673 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.159669 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:42885 (local address 127.13.94.190:46737)
0430 08:01:51.159810 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.159814 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.160399 (+ 585us) server_negotiation.cc:408] Connection header received
0430 08:01:51.161314 (+ 915us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.161318 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.161379 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.161485 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:51.162659 (+ 1174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.163175 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.163824 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.164008 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.166895 (+ 2887us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:51.166917 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:51.166920 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:51.166949 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:51.168559 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:51.169188 (+ 629us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:51.169193 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:51.169195 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:51.169276 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:51.169624 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:51.169627 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:51.169629 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:51.169810 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:51.169923 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.170444 (+ 521us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.170571 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260430 08:01:51.171368 24829 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.159886 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46737 (local address 127.13.94.129:42885)
0430 08:01:51.160263 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.160295 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:51.161112 (+ 817us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:51.161645 (+ 533us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:51.161654 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:51.162032 (+ 378us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:51.162497 (+ 465us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.162508 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.163298 (+ 790us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.163302 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:51.163703 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.163709 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.163926 (+ 217us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.165054 (+ 1128us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:51.165073 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:51.166738 (+ 1665us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:51.168685 (+ 1947us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:51.168691 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:51.168741 (+ 50us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:51.168994 (+ 253us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:51.169406 (+ 412us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:51.169409 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:51.169411 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:51.169508 (+ 97us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:51.169924 (+ 416us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:51.169932 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:51.170180 (+ 248us) client_negotiation.cc:770] Sending connection context
0430 08:01:51.170392 (+ 212us) client_negotiation.cc:241] Negotiation successful
0430 08:01:51.170637 (+ 245us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260430 08:01:51.172663 24827 heartbeater.cc:344] Connected to a master server at 127.13.94.190:46737
I20260430 08:01:51.172959 24827 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:51.173589 24827 heartbeater.cc:507] Master 127.13.94.190:46737 requested a full tablet report, sending...
I20260430 08:01:51.175175 24634 ts_manager.cc:194] Registered new tserver with Master: 11e8fa73361b4ab9ad1699235f74cc84 (127.13.94.129:41611)
I20260430 08:01:51.176322 24634 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:42885
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:51.220789 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:46737
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:51.324397 24834 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:51.324682 24834 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:51.324805 24834 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:51.328307 24834 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:51.328423 24834 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:51.328541 24834 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:01:51.333189 24834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:51.334391 24834 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:51.335304 24834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:51.342185 24842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:51.342139 24840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:51.342139 24839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:51.342522 24834 server_base.cc:1061] running on GCE node
I20260430 08:01:51.342901 24834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:51.343456 24834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:51.344637 24834 hybrid_clock.cc:648] HybridClock initialized: now 1777536111344627 us; error 31 us; skew 500 ppm
Apr 30 08:01:51 dist-test-slave-9bxp krb5kdc[24586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536111, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:51.347435 24834 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:01:51.348412 24834 webserver.cc:492] Webserver started at http://127.13.94.130:41131/ using document root <none> and password file <none>
I20260430 08:01:51.349011 24834 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:51.349061 24834 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:51.349229 24834 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:51.350864 24834 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "a9d5cad28f2849afa7eeeac2517119b8"
format_stamp: "Formatted at 2026-04-30 08:01:51 on dist-test-slave-9bxp"
server_key: "fed30c49a88e0886a4134c79868e2524"
server_key_iv: "b5d9f57b3b9f6ec00dddf532cb546b83"
server_key_version: "encryptionkey@0"
I20260430 08:01:51.351302 24834 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "a9d5cad28f2849afa7eeeac2517119b8"
format_stamp: "Formatted at 2026-04-30 08:01:51 on dist-test-slave-9bxp"
server_key: "fed30c49a88e0886a4134c79868e2524"
server_key_iv: "b5d9f57b3b9f6ec00dddf532cb546b83"
server_key_version: "encryptionkey@0"
I20260430 08:01:51.354687 24834 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:01:51.356861 24849 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:51.357802 24834 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260430 08:01:51.357928 24834 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "a9d5cad28f2849afa7eeeac2517119b8"
format_stamp: "Formatted at 2026-04-30 08:01:51 on dist-test-slave-9bxp"
server_key: "fed30c49a88e0886a4134c79868e2524"
server_key_iv: "b5d9f57b3b9f6ec00dddf532cb546b83"
server_key_version: "encryptionkey@0"
I20260430 08:01:51.358029 24834 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:51.371274 24834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:51.373801 24834 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:51.373994 24834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:51.374817 24834 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:51.375861 24834 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:51.375916 24834 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:51.375990 24834 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:51.376039 24834 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:51.386106 24834 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:35025
I20260430 08:01:51.386175 24962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:35025 every 8 connection(s)
I20260430 08:01:51.387200 24834 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
Apr 30 08:01:51 dist-test-slave-9bxp krb5kdc[24586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536111, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:51.397061 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 24834
I20260430 08:01:51.397167 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:01:51.397486 13690 external_mini_cluster.cc:1468] Setting key d4f9266382a422ac8e396653aca40f0e
I20260430 08:01:51.399511 24673 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.388991 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.130:36973 (local address 127.13.94.190:46737)
0430 08:01:51.389167 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.389172 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.389831 (+ 659us) server_negotiation.cc:408] Connection header received
0430 08:01:51.390612 (+ 781us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.390616 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.390664 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.390739 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:51.391894 (+ 1155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.392561 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.393213 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.393348 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.395858 (+ 2510us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:51.395874 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:51.395877 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:51.395902 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:51.397319 (+ 1417us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:51.397909 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:51.397915 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:51.397916 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:51.397970 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:51.398323 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:51.398326 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:51.398328 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:51.398517 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:51.398613 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.399267 (+ 654us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.399381 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260430 08:01:51.400440 24965 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.389285 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46737 (local address 127.13.94.130:36973)
0430 08:01:51.389698 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.389729 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:51.390437 (+ 708us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:51.390892 (+ 455us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:51.390899 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:51.391314 (+ 415us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:51.391770 (+ 456us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.391780 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.392669 (+ 889us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.392672 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:51.393108 (+ 436us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.393115 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.393258 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.394230 (+ 972us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:51.394247 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:51.395721 (+ 1474us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:51.397445 (+ 1724us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:51.397452 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:51.397463 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:51.397787 (+ 324us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:51.398087 (+ 300us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:51.398090 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:51.398094 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:51.398222 (+ 128us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:51.398618 (+ 396us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:51.398623 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:51.398903 (+ 280us) client_negotiation.cc:770] Sending connection context
0430 08:01:51.399226 (+ 323us) client_negotiation.cc:241] Negotiation successful
0430 08:01:51.399495 (+ 269us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260430 08:01:51.401585 24963 heartbeater.cc:344] Connected to a master server at 127.13.94.190:46737
I20260430 08:01:51.401854 24963 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:51.402364 24963 heartbeater.cc:507] Master 127.13.94.190:46737 requested a full tablet report, sending...
I20260430 08:01:51.403409 24634 ts_manager.cc:194] Registered new tserver with Master: a9d5cad28f2849afa7eeeac2517119b8 (127.13.94.130:35025)
I20260430 08:01:51.403959 24634 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.130@KRBTEST.COM'} at 127.13.94.130:36973
WARNING: no policy specified for kudu/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:01:51.449728 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.131:0
--local_ip_for_outbound_sockets=127.13.94.131
--webserver_interface=127.13.94.131
--webserver_port=0
--tserver_master_addrs=127.13.94.190:46737
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:01:51.558596 24970 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:01:51.558821 24970 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:01:51.558920 24970 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:01:51.562350 24970 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:01:51.562415 24970 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:01:51.562548 24970 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.131
I20260430 08:01:51.566875 24970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.94.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:46737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.131
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:01:51.567961 24970 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:01:51.568897 24970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:01:51.575359 24975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:01:51.575364 24978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:51.575481 24970 server_base.cc:1061] running on GCE node
W20260430 08:01:51.575605 24976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:01:51.575942 24970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:01:51.576475 24970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:01:51.577620 24970 hybrid_clock.cc:648] HybridClock initialized: now 1777536111577593 us; error 39 us; skew 500 ppm
Apr 30 08:01:51 dist-test-slave-9bxp krb5kdc[24586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536111, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:01:51.580396 24970 init.cc:377] Logged in from keytab as kudu/127.13.94.131@KRBTEST.COM (short username kudu)
I20260430 08:01:51.581393 24970 webserver.cc:492] Webserver started at http://127.13.94.131:33035/ using document root <none> and password file <none>
I20260430 08:01:51.581877 24970 fs_manager.cc:362] Metadata directory not provided
I20260430 08:01:51.581923 24970 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:01:51.582070 24970 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 08:01:51.583599 24970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/instance:
uuid: "f571afcbe99c4f5282a35a0ca73397ac"
format_stamp: "Formatted at 2026-04-30 08:01:51 on dist-test-slave-9bxp"
server_key: "0a2e927e5a5eba2686dcb649b68e6df0"
server_key_iv: "2440debaffbf7e39be7f750b2674d206"
server_key_version: "encryptionkey@0"
I20260430 08:01:51.584014 24970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance:
uuid: "f571afcbe99c4f5282a35a0ca73397ac"
format_stamp: "Formatted at 2026-04-30 08:01:51 on dist-test-slave-9bxp"
server_key: "0a2e927e5a5eba2686dcb649b68e6df0"
server_key_iv: "2440debaffbf7e39be7f750b2674d206"
server_key_version: "encryptionkey@0"
I20260430 08:01:51.587209 24970 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260430 08:01:51.590086 24985 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:51.591085 24970 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260430 08:01:51.591210 24970 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
uuid: "f571afcbe99c4f5282a35a0ca73397ac"
format_stamp: "Formatted at 2026-04-30 08:01:51 on dist-test-slave-9bxp"
server_key: "0a2e927e5a5eba2686dcb649b68e6df0"
server_key_iv: "2440debaffbf7e39be7f750b2674d206"
server_key_version: "encryptionkey@0"
I20260430 08:01:51.591315 24970 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:01:51.610466 24970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:01:51.613237 24970 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:01:51.613452 24970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:01:51.614048 24970 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:01:51.614966 24970 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:01:51.615039 24970 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:51.615123 24970 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:01:51.615176 24970 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:01:51.624306 24970 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.131:40717
I20260430 08:01:51.624328 25098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.131:40717 every 8 connection(s)
I20260430 08:01:51.625411 24970 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/data/info.pb
I20260430 08:01:51.625972 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 24970
I20260430 08:01:51.626104 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal/instance
I20260430 08:01:51.626394 13690 external_mini_cluster.cc:1468] Setting key 2004b8547074900cacf69c639ca447da
Apr 30 08:01:51 dist-test-slave-9bxp krb5kdc[24586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536111, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.131@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:01:51.638698 24673 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.627542 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:36075 (local address 127.13.94.190:46737)
0430 08:01:51.627698 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.627703 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.628550 (+ 847us) server_negotiation.cc:408] Connection header received
0430 08:01:51.629653 (+ 1103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.629657 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.629717 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.629826 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:51.631004 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.631552 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.632211 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.632411 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.635197 (+ 2786us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:51.635222 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:51.635224 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:51.635251 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:51.636612 (+ 1361us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:51.637220 (+ 608us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:51.637226 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:51.637233 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:51.637279 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:51.637580 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:51.637583 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:51.637585 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:51.637746 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:51.637853 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.638408 (+ 555us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.638532 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260430 08:01:51.639374 25101 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.627955 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:46737 (local address 127.13.94.131:36075)
0430 08:01:51.628422 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.628455 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:51.629273 (+ 818us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:51.629995 (+ 722us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:51.630004 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:51.630407 (+ 403us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:01:51.630854 (+ 447us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.630865 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.631681 (+ 816us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.631685 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:51.632090 (+ 405us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.632097 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.632310 (+ 213us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.633512 (+ 1202us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:01:51.633530 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:01:51.635030 (+ 1500us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:01:51.636759 (+ 1729us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:51.636765 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:51.636778 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:51.637040 (+ 262us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:51.637379 (+ 339us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:01:51.637383 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:01:51.637385 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:01:51.637479 (+ 94us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:01:51.637853 (+ 374us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:01:51.637860 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:01:51.638157 (+ 297us) client_negotiation.cc:770] Sending connection context
0430 08:01:51.638368 (+ 211us) client_negotiation.cc:241] Negotiation successful
0430 08:01:51.638597 (+ 229us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":134,"threads_started":1}
I20260430 08:01:51.640527 25099 heartbeater.cc:344] Connected to a master server at 127.13.94.190:46737
I20260430 08:01:51.640853 25099 heartbeater.cc:461] Registering TS with master...
I20260430 08:01:51.641387 25099 heartbeater.cc:507] Master 127.13.94.190:46737 requested a full tablet report, sending...
I20260430 08:01:51.642442 24634 ts_manager.cc:194] Registered new tserver with Master: f571afcbe99c4f5282a35a0ca73397ac (127.13.94.131:40717)
I20260430 08:01:51.643005 24634 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.131@KRBTEST.COM'} at 127.13.94.131:36075
I20260430 08:01:51.653003 13690 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260430 08:01:51.661628 24673 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.654922 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.13.94.190:46737)
0430 08:01:51.655095 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.655099 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.655262 (+ 163us) server_negotiation.cc:408] Connection header received
0430 08:01:51.655440 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.655446 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.655497 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.655582 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:01:51.656422 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.656977 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.657624 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.657830 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.658844 (+ 1014us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:01:51.658860 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:01:51.658863 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:01:51.658893 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:01:51.660079 (+ 1186us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:51.660474 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:51.660478 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:51.660480 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:51.660528 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:01:51.660814 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:01:51.660818 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:01:51.660819 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:01:51.660986 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:01:51.661124 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.661352 (+ 228us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.661468 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260430 08:01:51.663991 24634 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 08:01:51.666414 24634 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 08:01:51.678913 25113 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.674960 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.131:40717 (local address 127.0.0.1:50248)
0430 08:01:51.675673 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.675686 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:51.675766 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:51.676371 (+ 605us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:51.676374 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:51.676391 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:51.676597 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.676603 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.677625 (+ 1022us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.677629 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:51.678498 (+ 869us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.678507 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.678618 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.678634 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 08:01:51.678688 (+ 54us) client_negotiation.cc:241] Negotiation successful
0430 08:01:51.678746 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":85,"threads_started":1}
I20260430 08:01:51.679760 25111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.674497 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.13.94.129:41611)
0430 08:01:51.674850 (+ 353us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.674856 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.675246 (+ 390us) server_negotiation.cc:408] Connection header received
0430 08:01:51.675343 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.675347 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.675530 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.675665 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:51.676284 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.677374 (+ 1090us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.678816 (+ 1442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.679353 (+ 537us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.679472 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.679541 (+ 69us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.679630 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":91,"threads_started":1}
I20260430 08:01:51.679795 25114 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.675029 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.13.94.130:35025)
0430 08:01:51.675351 (+ 322us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.675357 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.675803 (+ 446us) server_negotiation.cc:408] Connection header received
0430 08:01:51.675888 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.675893 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.676073 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.676218 (+ 145us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:51.676624 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.677749 (+ 1125us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.678988 (+ 1239us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.679478 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.679552 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.679604 (+ 52us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.679670 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":152,"threads_started":1}
I20260430 08:01:51.679021 25112 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.674538 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:35025 (local address 127.0.0.1:60712)
0430 08:01:51.675619 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.675633 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:51.675711 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:51.676218 (+ 507us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:51.676221 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:51.676239 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:51.676496 (+ 257us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.676503 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.677854 (+ 1351us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.677858 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:51.678666 (+ 808us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.678673 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.678761 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.678777 (+ 16us) client_negotiation.cc:770] Sending connection context
0430 08:01:51.678823 (+ 46us) client_negotiation.cc:241] Negotiation successful
0430 08:01:51.678865 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":104,"threads_started":1}
I20260430 08:01:51.680024 25115 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.675128 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.13.94.131:40717)
0430 08:01:51.675531 (+ 403us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.675537 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.675862 (+ 325us) server_negotiation.cc:408] Connection header received
0430 08:01:51.675947 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.675951 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.676104 (+ 153us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.676245 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:51.676733 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.677502 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.679279 (+ 1777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.679648 (+ 369us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.679718 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.679768 (+ 50us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.679858 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":135,"threads_started":1}
I20260430 08:01:51.678913 25110 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.674304 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:41611 (local address 127.0.0.1:51726)
0430 08:01:51.674922 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.674963 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:51.675107 (+ 144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:51.675831 (+ 724us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:51.675838 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:51.675859 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:51.676140 (+ 281us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.676147 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.677499 (+ 1352us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.677503 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:51.678384 (+ 881us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.678393 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.678497 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.678547 (+ 50us) client_negotiation.cc:770] Sending connection context
0430 08:01:51.678624 (+ 77us) client_negotiation.cc:241] Negotiation successful
0430 08:01:51.678691 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":99,"threads_started":1}
I20260430 08:01:51.682933 24761 tablet_service.cc:1511] Processing CreateTablet for tablet 1350ebfe11bd48369ea8cbe33ba4ceee (DEFAULT_TABLE table=test-table [id=3d78776c061b4a1f8434d0c554e85975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:51.682946 24897 tablet_service.cc:1511] Processing CreateTablet for tablet 1350ebfe11bd48369ea8cbe33ba4ceee (DEFAULT_TABLE table=test-table [id=3d78776c061b4a1f8434d0c554e85975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:51.682947 25033 tablet_service.cc:1511] Processing CreateTablet for tablet 1350ebfe11bd48369ea8cbe33ba4ceee (DEFAULT_TABLE table=test-table [id=3d78776c061b4a1f8434d0c554e85975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260430 08:01:51.683836 24761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1350ebfe11bd48369ea8cbe33ba4ceee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:51.683851 24897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1350ebfe11bd48369ea8cbe33ba4ceee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:51.683851 25033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1350ebfe11bd48369ea8cbe33ba4ceee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:01:51.689047 25116 tablet_bootstrap.cc:492] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84: Bootstrap starting.
I20260430 08:01:51.690304 25117 tablet_bootstrap.cc:492] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: Bootstrap starting.
I20260430 08:01:51.690378 25118 tablet_bootstrap.cc:492] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8: Bootstrap starting.
I20260430 08:01:51.691401 25116 tablet_bootstrap.cc:654] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:51.692271 25116 log.cc:826] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:51.692320 25117 tablet_bootstrap.cc:654] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:51.692440 25118 tablet_bootstrap.cc:654] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8: Neither blocks nor log segments found. Creating new log.
I20260430 08:01:51.693199 25118 log.cc:826] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:51.693213 25117 log.cc:826] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: Log is configured to *not* fsync() on all Append() calls
I20260430 08:01:51.694300 25116 tablet_bootstrap.cc:492] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84: No bootstrap required, opened a new log
I20260430 08:01:51.694525 25116 ts_tablet_manager.cc:1403] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260430 08:01:51.694886 25118 tablet_bootstrap.cc:492] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8: No bootstrap required, opened a new log
I20260430 08:01:51.694886 25117 tablet_bootstrap.cc:492] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: No bootstrap required, opened a new log
I20260430 08:01:51.695067 25117 ts_tablet_manager.cc:1403] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260430 08:01:51.695078 25118 ts_tablet_manager.cc:1403] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260430 08:01:51.697146 25116 raft_consensus.cc:359] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } }
I20260430 08:01:51.697351 25116 raft_consensus.cc:385] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:51.697448 25116 raft_consensus.cc:740] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e8fa73361b4ab9ad1699235f74cc84, State: Initialized, Role: FOLLOWER
I20260430 08:01:51.697890 25116 consensus_queue.cc:260] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } }
I20260430 08:01:51.698567 24827 heartbeater.cc:499] Master 127.13.94.190:46737 was elected leader, sending a full tablet report...
I20260430 08:01:51.698812 25116 ts_tablet_manager.cc:1434] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260430 08:01:51.698722 25118 raft_consensus.cc:359] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } }
I20260430 08:01:51.698910 25118 raft_consensus.cc:385] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:51.699015 25118 raft_consensus.cc:740] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d5cad28f2849afa7eeeac2517119b8, State: Initialized, Role: FOLLOWER
I20260430 08:01:51.699433 25118 consensus_queue.cc:260] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } }
I20260430 08:01:51.699550 25117 raft_consensus.cc:359] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } }
I20260430 08:01:51.699774 25117 raft_consensus.cc:385] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:01:51.699860 25117 raft_consensus.cc:740] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f571afcbe99c4f5282a35a0ca73397ac, State: Initialized, Role: FOLLOWER
I20260430 08:01:51.700191 24963 heartbeater.cc:499] Master 127.13.94.190:46737 was elected leader, sending a full tablet report...
I20260430 08:01:51.700368 25118 ts_tablet_manager.cc:1434] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260430 08:01:51.700289 25117 consensus_queue.cc:260] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } }
I20260430 08:01:51.700933 25099 heartbeater.cc:499] Master 127.13.94.190:46737 was elected leader, sending a full tablet report...
I20260430 08:01:51.701519 25117 ts_tablet_manager.cc:1434] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260430 08:01:51.703331 25124 raft_consensus.cc:493] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 08:01:51.703481 25124 raft_consensus.cc:515] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } }
I20260430 08:01:51.704617 25124 leader_election.cc:290] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11e8fa73361b4ab9ad1699235f74cc84 (127.13.94.129:41611), a9d5cad28f2849afa7eeeac2517119b8 (127.13.94.130:35025)
I20260430 08:01:51.707290 25101 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.704785 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:41611 (local address 127.13.94.131:32977)
0430 08:01:51.704934 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.704947 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:51.705045 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:51.705375 (+ 330us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:51.705378 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:51.705394 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:51.705578 (+ 184us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.705583 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.706319 (+ 736us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.706324 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:51.706902 (+ 578us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.706909 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.707043 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.707060 (+ 17us) client_negotiation.cc:770] Sending connection context
0430 08:01:51.707113 (+ 53us) client_negotiation.cc:241] Negotiation successful
0430 08:01:51.707168 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260430 08:01:51.707569 25125 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.704868 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:35025 (local address 127.13.94.131:57999)
0430 08:01:51.705222 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.705237 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0430 08:01:51.705347 (+ 110us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:01:51.705605 (+ 258us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:01:51.705608 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:01:51.705625 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0430 08:01:51.705856 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.705861 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.706583 (+ 722us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.706587 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:01:51.707173 (+ 586us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:01:51.707179 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.707307 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.707322 (+ 15us) client_negotiation.cc:770] Sending connection context
0430 08:01:51.707372 (+ 50us) client_negotiation.cc:241] Negotiation successful
0430 08:01:51.707431 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":98,"threads_started":1}
I20260430 08:01:51.707787 25111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.704931 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:32977 (local address 127.13.94.129:41611)
0430 08:01:51.705063 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.705069 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.705088 (+ 19us) server_negotiation.cc:408] Connection header received
0430 08:01:51.705140 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.705143 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.705193 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.705263 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:51.705691 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.706204 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.707031 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.707529 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.707561 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.707614 (+ 53us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.707671 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260430 08:01:51.708078 25114 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.705269 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.131:57999 (local address 127.13.94.130:35025)
0430 08:01:51.705420 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.705426 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.705444 (+ 18us) server_negotiation.cc:408] Connection header received
0430 08:01:51.705476 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.705479 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.705512 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.705594 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0430 08:01:51.705986 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.706468 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.707292 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.707788 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.707823 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.707903 (+ 80us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.707964 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260430 08:01:51.708678 24781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1350ebfe11bd48369ea8cbe33ba4ceee" candidate_uuid: "f571afcbe99c4f5282a35a0ca73397ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e8fa73361b4ab9ad1699235f74cc84" is_pre_election: true
I20260430 08:01:51.708788 24917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1350ebfe11bd48369ea8cbe33ba4ceee" candidate_uuid: "f571afcbe99c4f5282a35a0ca73397ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d5cad28f2849afa7eeeac2517119b8" is_pre_election: true
I20260430 08:01:51.708988 24781 raft_consensus.cc:2468] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f571afcbe99c4f5282a35a0ca73397ac in term 0.
I20260430 08:01:51.709021 24917 raft_consensus.cc:2468] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f571afcbe99c4f5282a35a0ca73397ac in term 0.
I20260430 08:01:51.709468 24988 leader_election.cc:304] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e8fa73361b4ab9ad1699235f74cc84, f571afcbe99c4f5282a35a0ca73397ac; no voters:
I20260430 08:01:51.709719 25124 raft_consensus.cc:2804] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 08:01:51.709787 25124 raft_consensus.cc:493] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 08:01:51.709828 25124 raft_consensus.cc:3060] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:51.710713 25124 raft_consensus.cc:515] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } }
I20260430 08:01:51.711064 25124 leader_election.cc:290] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [CANDIDATE]: Term 1 election: Requested vote from peers 11e8fa73361b4ab9ad1699235f74cc84 (127.13.94.129:41611), a9d5cad28f2849afa7eeeac2517119b8 (127.13.94.130:35025)
I20260430 08:01:51.711359 24781 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1350ebfe11bd48369ea8cbe33ba4ceee" candidate_uuid: "f571afcbe99c4f5282a35a0ca73397ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e8fa73361b4ab9ad1699235f74cc84"
I20260430 08:01:51.711498 24781 raft_consensus.cc:3060] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:51.711452 24917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1350ebfe11bd48369ea8cbe33ba4ceee" candidate_uuid: "f571afcbe99c4f5282a35a0ca73397ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d5cad28f2849afa7eeeac2517119b8"
I20260430 08:01:51.711625 24917 raft_consensus.cc:3060] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:01:51.712474 24781 raft_consensus.cc:2468] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f571afcbe99c4f5282a35a0ca73397ac in term 1.
I20260430 08:01:51.712604 24917 raft_consensus.cc:2468] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f571afcbe99c4f5282a35a0ca73397ac in term 1.
I20260430 08:01:51.712786 24988 leader_election.cc:304] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11e8fa73361b4ab9ad1699235f74cc84, f571afcbe99c4f5282a35a0ca73397ac; no voters:
I20260430 08:01:51.712996 25124 raft_consensus.cc:2804] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:01:51.713212 25124 raft_consensus.cc:697] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 1 LEADER]: Becoming Leader. State: Replica: f571afcbe99c4f5282a35a0ca73397ac, State: Running, Role: LEADER
I20260430 08:01:51.713532 25124 consensus_queue.cc:237] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } }
I20260430 08:01:51.716364 24632 catalog_manager.cc:5671] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac reported cstate change: term changed from 0 to 1, leader changed from <none> to f571afcbe99c4f5282a35a0ca73397ac (127.13.94.131). New cstate: current_term: 1 leader_uuid: "f571afcbe99c4f5282a35a0ca73397ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f571afcbe99c4f5282a35a0ca73397ac" member_type: VOTER last_known_addr { host: "127.13.94.131" port: 40717 } health_report { overall_health: HEALTHY } } }
I20260430 08:01:51.735105 25115 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.732355 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.13.94.131:40717)
0430 08:01:51.732495 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0430 08:01:51.732499 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0430 08:01:51.732511 (+ 12us) server_negotiation.cc:408] Connection header received
0430 08:01:51.732581 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:01:51.732584 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:01:51.732635 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:01:51.732784 (+ 149us) server_negotiation.cc:227] Negotiated authn=TOKEN
0430 08:01:51.733076 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.733566 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:01:51.734193 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:01:51.734328 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:01:51.734425 (+ 97us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0430 08:01:51.734773 (+ 348us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0430 08:01:51.734860 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0430 08:01:51.734959 (+ 99us) server_negotiation.cc:300] Negotiation successful
0430 08:01:51.734999 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260430 08:01:51.740754 24781 raft_consensus.cc:1275] T 1350ebfe11bd48369ea8cbe33ba4ceee P 11e8fa73361b4ab9ad1699235f74cc84 [term 1 FOLLOWER]: Refusing update from remote peer f571afcbe99c4f5282a35a0ca73397ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:51.740777 24917 raft_consensus.cc:1275] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [term 1 FOLLOWER]: Refusing update from remote peer f571afcbe99c4f5282a35a0ca73397ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 08:01:51.741480 25126 consensus_queue.cc:1048] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "11e8fa73361b4ab9ad1699235f74cc84" member_type: VOTER last_known_addr { host: "127.13.94.129" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:51.741673 25124 consensus_queue.cc:1048] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9d5cad28f2849afa7eeeac2517119b8" member_type: VOTER last_known_addr { host: "127.13.94.130" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 08:01:51.750440 25131 mvcc.cc:204] Tried to move back new op lower bound from 7280787913682780160 to 7280787913581182976. Current Snapshot: MvccSnapshot[applied={T|T < 7280787913682780160}]
I20260430 08:01:51.758827 24633 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:51.758976 24633 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260430 08:01:51.761504 24633 catalog_manager.cc:5958] T 00000000000000000000000000000000 P e7ccd203f9bc42afb1cae3f7a766b5cd: Sending DeleteTablet for 3 replicas of tablet 1350ebfe11bd48369ea8cbe33ba4ceee
I20260430 08:01:51.762409 24761 tablet_service.cc:1558] Processing DeleteTablet for tablet 1350ebfe11bd48369ea8cbe33ba4ceee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:51 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:51726
I20260430 08:01:51.762406 24897 tablet_service.cc:1558] Processing DeleteTablet for tablet 1350ebfe11bd48369ea8cbe33ba4ceee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:51 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:60712
I20260430 08:01:51.762663 25033 tablet_service.cc:1558] Processing DeleteTablet for tablet 1350ebfe11bd48369ea8cbe33ba4ceee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-30 08:01:51 UTC) from {username='kudu', principal='kudu/127.13.94.190@KRBTEST.COM'} at 127.0.0.1:50248
I20260430 08:01:51.763130 25139 tablet_replica.cc:333] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8: stopping tablet replica
I20260430 08:01:51.763356 25141 tablet_replica.cc:333] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: stopping tablet replica
I20260430 08:01:51.763429 25139 raft_consensus.cc:2243] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 08:01:51.763514 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 24698
I20260430 08:01:51.763724 25141 raft_consensus.cc:2243] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 1 LEADER]: Raft consensus shutting down.
I20260430 08:01:51.764129 25141 raft_consensus.cc:2272] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:51.764330 25139 raft_consensus.cc:2272] T 1350ebfe11bd48369ea8cbe33ba4ceee P a9d5cad28f2849afa7eeeac2517119b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 08:01:51.765793 25141 ts_tablet_manager.cc:1916] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260430 08:01:51.768049 25141 ts_tablet_manager.cc:1929] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260430 08:01:51.768164 25141 log.cc:1199] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: Deleting WAL directory at /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777536011314967-13690-0/minicluster-data/ts-2/wal/wals/1350ebfe11bd48369ea8cbe33ba4ceee
I20260430 08:01:51.768469 25141 ts_tablet_manager.cc:1950] T 1350ebfe11bd48369ea8cbe33ba4ceee P f571afcbe99c4f5282a35a0ca73397ac: Deleting consensus metadata
I20260430 08:01:51.769795 24620 catalog_manager.cc:5002] TS f571afcbe99c4f5282a35a0ca73397ac (127.13.94.131:40717): tablet 1350ebfe11bd48369ea8cbe33ba4ceee (table test-table [id=3d78776c061b4a1f8434d0c554e85975]) successfully deleted
W20260430 08:01:51.772657 24620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.94.129:41611 (error 108)
I20260430 08:01:51.772850 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 24834
I20260430 08:01:51.774307 25110 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.773830 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.129:41611 (local address 127.0.0.1:51740)
0430 08:01:51.774091 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.774196 (+ 105us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.129:41611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260430 08:01:51.774919 24620 catalog_manager.cc:4729] TS 11e8fa73361b4ab9ad1699235f74cc84 (127.13.94.129:41611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1350ebfe11bd48369ea8cbe33ba4ceee: Network error: Client connection negotiation failed: client connection to 127.13.94.129:41611: connect: Connection refused (error 111)
I20260430 08:01:51.780548 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 24970
I20260430 08:01:51.780805 25110 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:01:51.780460 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.130:35025 (local address 127.0.0.1:60724)
0430 08:01:51.780620 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0430 08:01:51.780675 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.130:35025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260430 08:01:51.787376 13690 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu with pid 24602
2026-04-30T08:01:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3255 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 30 08:01:51 dist-test-slave-9bxp krb5kdc[25145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 30 08:01:51 dist-test-slave-9bxp krb5kdc[25145](info): set up 2 sockets
Apr 30 08:01:51 dist-test-slave-9bxp krb5kdc[25145](info): commencing operation
krb5kdc: starting...
W20260430 08:01:53.789695 13690 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/test-user.keytab.
Apr 30 08:01:53 dist-test-slave-9bxp krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-30T08:01:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-30T08:01:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.94.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.94.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.94.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/rangeradmin_127.13.94.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.94.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/rangeradmin_127.13.94.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.94.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.94.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.94.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/rangerlookup_127.13.94.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.94.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/rangerlookup_127.13.94.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.94.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/HTTP_127.13.94.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.94.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/HTTP_127.13.94.148@KRBTEST.COM.keytab.
I20260430 08:01:53.959966 13690 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWlZwuK/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/postgres -l logfile start
2026-04-30 08:01:56.762 UTC [25176] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-04-30 08:01:56.762 UTC [25176] LOG: listening on IPv4 address "127.13.94.148", port 56665
2026-04-30 08:01:56.767 UTC [25176] LOG: listening on Unix socket "/tmp/.s.PGSQL.56665"
2026-04-30 08:01:56.773 UTC [25181] LOG: database system was shut down at 2026-04-30 08:01:55 UTC
2026-04-30 08:01:56.777 UTC [25176] LOG: database system is ready to accept connections
I20260430 08:01:58.693710 13690 mini_postgres.cc:96] Postgres bound to 56665
2026-04-30 08:01:58.699 UTC [25187] FATAL: database "slave" does not exist
127.13.94.148:56665 - accepting connections
I20260430 08:01:58.700489 13690 mini_ranger.cc:161] Starting Ranger...
I20260430 08:01:58.718657 13690 mini_ranger.cc:84] Created miniranger Postgres user
I20260430 08:01:58.790705 13690 mini_ranger.cc:87] Created ranger Postgres database
I20260430 08:01:58.790817 13690 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-admin
2026-04-30 08:01:59,088 [I] DB FLAVOR :POSTGRES
2026-04-30 08:01:59,089 [I] --------- Verifying Ranger DB connection ---------
2026-04-30 08:01:59,089 [I] Checking connection..
2026-04-30 08:01:59,089 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-04-30 08:01:59,418 [I] Checking connection passed.
2026-04-30 08:01:59,419 [I] --------- Verifying version history table ---------
2026-04-30 08:01:59,419 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-30 08:01:59,770 [I] Table x_db_version_h does not exist in database ranger
2026-04-30 08:01:59,770 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-30 08:02:00,132 [I] Table x_db_version_h does not exist in database ranger
2026-04-30 08:02:00,132 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-04-30 08:02:00,133 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-04-30 08:02:00.481 UTC [25301] WARNING: there is no transaction in progress
2026-04-30 08:02:00,506 [I] create_dbversion_catalog.sql file imported successfully
2026-04-30 08:02:00,507 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-30 08:02:00,835 [I] Table x_db_version_h already exists in database 'ranger'
2026-04-30 08:02:00,835 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-30 08:02:00,835 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-04-30 08:02:01,167 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-04-30 08:02:01,527 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9bxp.c.gcp-upstream.internal','N') ;"
2026-04-30 08:02:01,842 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-30 08:02:02,156 [I] Table x_portal_user does not exist in database ranger
2026-04-30 08:02:02,157 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-30 08:02:02,505 [I] Table x_policy_ref_group does not exist in database ranger
2026-04-30 08:02:02,505 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-30 08:02:02,857 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-30 08:02:03,194 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-04-30 08:02:03,194 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-04-30 08:02:04.118 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.129 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.142 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.154 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.165 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.177 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.229 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.235 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.245 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.255 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.267 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.277 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.286 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.291 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.295 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04.299 UTC [25501] WARNING: there is no transaction in progress
2026-04-30 08:02:04,765 [I] ranger_core_db_postgres.sql file imported successfully
2026-04-30 08:02:04,766 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-04-30 08:02:05,100 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-04-30 08:02:05,101 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-30 08:02:05,404 [I] Table x_portal_user already exists in database 'ranger'
2026-04-30 08:02:05,404 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-30 08:02:05,755 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-04-30 08:02:05,756 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-30 08:02:06,085 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-30 08:02:06,424 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9bxp.c.gcp-upstream.internal';"
2026-04-30 08:02:06,752 [I] CORE_DB_SCHEMA import status has been updated
2026-04-30 08:02:06,753 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-04-30 08:02:07,086 [I] DB_PATCHES have already been applied
I20260430 08:02:07.091764 13690 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskWlZwuK/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260430 08:02:07.091832 13690 mini_ranger.cc:193] Using host: 127.13.94.148
I20260430 08:02:07.094995 13690 mini_ranger.cc:239] Ranger admin URL: http://127.13.94.148:47069
Apr 30, 2026 8:02:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 30, 2026 8:02:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 30, 2026 8:02:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-admin/ews/webapp
Apr 30, 2026 8:02:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Apr 30, 2026 8:02:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-admin/ews/webapp] .....
Apr 30, 2026 8:02:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-admin/ews/webapp].
Apr 30, 2026 8:02:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260430 08:02:09.388753 13690 mini_ranger.cc:160] Time spent starting Ranger: real 10.688s user 0.004s sys 0.007s
Apr 30, 2026 8:02:32 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Apr 30, 2026 8:02:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
Apr 30, 2026 8:02:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Apr 30, 2026 8:02:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Apr 30, 2026 8:02:33 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260430 08:02:34.270763 13690 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.94.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.94.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.94.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/rangerkms_127.13.94.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.94.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/rangerkms_127.13.94.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.94.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.94.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/HTTP_127.13.94.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.94.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/HTTP_127.13.94.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Apr 30 08:02:34 dist-test-slave-9bxp krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536154, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260430 08:02:34.380331 13690 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260430 08:02:34.392984 13690 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260430 08:02:34.521374 13690 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260430 08:02:34.521469 13690 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-kms
I20260430 08:02:34.521507 13690 mini_ranger_kms.cc:227] Using postgres at 127.13.94.148:56665
2026-04-30 08:02:34,719 [I] DB FLAVOR :POSTGRES
2026-04-30 08:02:34,719 [I] --------- Verifying Ranger DB connection ---------
2026-04-30 08:02:34,719 [I] Checking connection
2026-04-30 08:02:34,720 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-04-30 08:02:35,056 [I] connection success
2026-04-30 08:02:35,057 [I] --------- Verifying Ranger DB tables ---------
2026-04-30 08:02:35,057 [I] Verifying table ranger_masterkey in database rangerkms
2026-04-30 08:02:35,057 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-04-30 08:02:35,353 [I] Table ranger_masterkey does not exist in database rangerkms
2026-04-30 08:02:35,353 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-30 08:02:35,353 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-04-30 08:02:35,353 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.94.148:56665/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-04-30 08:02:35,681 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260430 08:02:36.262434 13690 mini_ranger_kms.cc:326] Created kms service
I20260430 08:02:36.485344 13690 mini_ranger_kms.cc:342] Created kudu user
I20260430 08:02:36.538503 13690 mini_ranger_kms.cc:359] Created rangerkms user
Apr 30 08:02:36 dist-test-slave-9bxp krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536154, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.94.148@KRBTEST.COM
I20260430 08:02:37.015904 13690 mini_ranger_kms.cc:400] Added ranger policy
I20260430 08:02:37.016078 13690 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWlZwuK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWlZwuK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260430 08:02:37.016103 13690 mini_ranger_kms.cc:240] Using host: 127.13.94.148
I20260430 08:02:37.019863 13690 mini_ranger_kms.cc:292] Ranger KMS PID: 25801
I20260430 08:02:37.019977 13690 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.13.94.148:38483
Apr 30, 2026 8:02:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 30, 2026 8:02:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 30, 2026 8:02:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Apr 30, 2026 8:02:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-kms/ews/webapp] .....
Apr 30, 2026 8:02:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-kms/ews/webapp].
Apr 30, 2026 8:02:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260430 08:02:39.357890 13690 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.978s user 0.024s sys 0.158s
I20260430 08:02:39.358038 13690 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260430 08:02:39.358122 13690 mini_ranger_kms.cc:417] 127.13.94.148:38483/kms/v1/keys
Apr 30 08:02:39 dist-test-slave-9bxp krb5kdc[25145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777536159, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.94.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Apr 30 08:02:40 dist-test-slave-9bxp krb5kdc[25145](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777536159, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.94.148@KRBTEST.COM for HTTP/127.13.94.148@KRBTEST.COM
Apr 30 08:02:40 dist-test-slave-9bxp krb5kdc[25145](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777536159, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.94.148@KRBTEST.COM for HTTP/127.13.94.148@KRBTEST.COM
Apr 30 08:02:43 dist-test-slave-9bxp krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:02:43.179831 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:39047
--webserver_interface=127.13.94.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:38851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.94.190:39047
--ranger_config_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.94.148:38483/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:02:43.281975 25868 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:02:43.282203 25868 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:02:43.282253 25868 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:02:43.285456 25868 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260430 08:02:43.285511 25868 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:02:43.285532 25868 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:02:43.285550 25868 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260430 08:02:43.285567 25868 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260430 08:02:43.289868 25868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:38851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.94.148:38483/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.94.190:39047
--ranger_config_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.94.190:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.94.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:02:43.290882 25868 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:02:43.291625 25868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:02:43.297212 25873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:02:43.297250 25874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:02:43.297446 25868 server_base.cc:1061] running on GCE node
W20260430 08:02:43.297250 25876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:02:43.297966 25868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:02:43.298817 25868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:02:43.300019 25868 hybrid_clock.cc:648] HybridClock initialized: now 1777536163299983 us; error 48 us; skew 500 ppm
Apr 30 08:02:43 dist-test-slave-9bxp krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:02:43.302827 25868 init.cc:377] Logged in from keytab as kudu/127.13.94.190@KRBTEST.COM (short username kudu)
I20260430 08:02:43.303934 25868 webserver.cc:492] Webserver started at http://127.13.94.190:37167/ using document root <none> and password file <none>
I20260430 08:02:43.304546 25868 fs_manager.cc:362] Metadata directory not provided
I20260430 08:02:43.304623 25868 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:02:43.304908 25868 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 30 08:02:43 dist-test-slave-9bxp krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.190@KRBTEST.COM for HTTP/127.13.94.148@KRBTEST.COM
I20260430 08:02:43.354336 25868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/data/instance:
uuid: "420d706cfc664c348f008e9dd13359f0"
format_stamp: "Formatted at 2026-04-30 08:02:43 on dist-test-slave-9bxp"
server_key: "c125977c33c94f5af77dc53cc1290032"
server_key_iv: "1931d788bcc673fbd9474be5068acbdb"
server_key_version: "kuduclusterkey@0"
I20260430 08:02:43.354902 25868 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/wal/instance:
uuid: "420d706cfc664c348f008e9dd13359f0"
format_stamp: "Formatted at 2026-04-30 08:02:43 on dist-test-slave-9bxp"
server_key: "c125977c33c94f5af77dc53cc1290032"
server_key_iv: "1931d788bcc673fbd9474be5068acbdb"
server_key_version: "kuduclusterkey@0"
I20260430 08:02:43.358446 25868 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260430 08:02:43.374526 25884 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:02:43.375759 25868 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260430 08:02:43.376012 25868 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/wal
uuid: "420d706cfc664c348f008e9dd13359f0"
format_stamp: "Formatted at 2026-04-30 08:02:43 on dist-test-slave-9bxp"
server_key: "c125977c33c94f5af77dc53cc1290032"
server_key_iv: "1931d788bcc673fbd9474be5068acbdb"
server_key_version: "kuduclusterkey@0"
I20260430 08:02:43.376158 25868 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:02:43.388684 25868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:02:43.394421 25868 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:02:43.394627 25868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:02:43.402724 25868 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.190:39047
I20260430 08:02:43.402806 25936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.190:39047 every 8 connection(s)
I20260430 08:02:43.403882 25868 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/data/info.pb
I20260430 08:02:43.406617 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 25868
I20260430 08:02:43.406749 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/wal/instance
I20260430 08:02:43.406831 25937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 08:02:43.407037 13690 external_mini_cluster.cc:1468] Setting key eb0fbd5619e36570dd57ef16eb032a18
I20260430 08:02:43.413120 25937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0: Bootstrap starting.
I20260430 08:02:43.415758 25937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0: Neither blocks nor log segments found. Creating new log.
I20260430 08:02:43.416661 25937 log.cc:826] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0: Log is configured to *not* fsync() on all Append() calls
Apr 30 08:02:43 dist-test-slave-9bxp krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:02:43.419342 25937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0: No bootstrap required, opened a new log
I20260430 08:02:43.422760 25937 raft_consensus.cc:359] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d706cfc664c348f008e9dd13359f0" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 39047 } }
I20260430 08:02:43.423038 25937 raft_consensus.cc:385] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 08:02:43.423113 25937 raft_consensus.cc:740] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420d706cfc664c348f008e9dd13359f0, State: Initialized, Role: FOLLOWER
I20260430 08:02:43.423632 25937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d706cfc664c348f008e9dd13359f0" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 39047 } }
I20260430 08:02:43.423820 25937 raft_consensus.cc:399] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 08:02:43.423936 25937 raft_consensus.cc:493] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 08:02:43.424067 25937 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [term 0 FOLLOWER]: Advancing to term 1
I20260430 08:02:43.425215 25937 raft_consensus.cc:515] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d706cfc664c348f008e9dd13359f0" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 39047 } }
I20260430 08:02:43.425602 25937 leader_election.cc:304] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 420d706cfc664c348f008e9dd13359f0; no voters:
I20260430 08:02:43.425945 25937 leader_election.cc:290] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 08:02:43.426008 25940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:43.408983 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.13.94.190:39047)
0430 08:02:43.411777 (+ 2794us) server_negotiation.cc:207] Beginning negotiation
0430 08:02:43.411789 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0430 08:02:43.411826 (+ 37us) server_negotiation.cc:408] Connection header received
0430 08:02:43.412493 (+ 667us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:02:43.412519 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:02:43.412947 (+ 428us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:02:43.414518 (+ 1571us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:02:43.414864 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:02:43.416070 (+ 1206us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:02:43.417274 (+ 1204us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:02:43.417547 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:02:43.420075 (+ 2528us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:02:43.420098 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:02:43.420115 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:02:43.420153 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:02:43.422992 (+ 2839us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:02:43.423640 (+ 648us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:02:43.423649 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:02:43.423657 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:02:43.423741 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:02:43.424068 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:02:43.424073 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:02:43.424078 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:02:43.424495 (+ 417us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:02:43.424690 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
0430 08:02:43.425031 (+ 341us) server_negotiation.cc:300] Negotiation successful
0430 08:02:43.425256 (+ 225us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2652,"thread_start_us":136,"threads_started":1}
I20260430 08:02:43.426165 25942 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 08:02:43.426399 25942 raft_consensus.cc:697] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [term 1 LEADER]: Becoming Leader. State: Replica: 420d706cfc664c348f008e9dd13359f0, State: Running, Role: LEADER
I20260430 08:02:43.426728 25942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d706cfc664c348f008e9dd13359f0" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 39047 } }
I20260430 08:02:43.427140 25937 sys_catalog.cc:565] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 08:02:43.428885 25944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 420d706cfc664c348f008e9dd13359f0. Latest consensus state: current_term: 1 leader_uuid: "420d706cfc664c348f008e9dd13359f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d706cfc664c348f008e9dd13359f0" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 39047 } } }
I20260430 08:02:43.428920 25943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "420d706cfc664c348f008e9dd13359f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420d706cfc664c348f008e9dd13359f0" member_type: VOTER last_known_addr { host: "127.13.94.190" port: 39047 } } }
I20260430 08:02:43.429025 25944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [sys.catalog]: This master's current role is: LEADER
I20260430 08:02:43.429028 25943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0 [sys.catalog]: This master's current role is: LEADER
I20260430 08:02:43.430106 25937 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260430 08:02:43.430311 25951 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 08:02:43.436113 25951 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 08:02:43.443024 25951 catalog_manager.cc:1357] Generated new cluster ID: 7308795779ed48f78e5967ac2566a60a
I20260430 08:02:43.443096 25951 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 08:02:43.457159 25951 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 08:02:43.458412 25951 catalog_manager.cc:1514] Loading token signing keys...
I20260430 08:02:43.467645 25951 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 420d706cfc664c348f008e9dd13359f0: Generated new TSK 0
I20260430 08:02:43.468395 25951 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.94.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:02:44.230341 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.94.148:38483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.129:0
--local_ip_for_outbound_sockets=127.13.94.129
--webserver_interface=127.13.94.129
--webserver_port=0
--tserver_master_addrs=127.13.94.190:39047
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:38851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260430 08:02:44.367321 25996 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:02:44.367738 25996 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:02:44.367879 25996 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:02:44.373066 25996 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:02:44.373241 25996 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:02:44.373413 25996 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.129
I20260430 08:02:44.379964 25996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:38851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.94.148:38483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.94.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:39047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.129
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:02:44.381619 25996 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:02:44.382843 25996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:02:44.391311 26003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:02:44.391381 26006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:02:44.391646 26004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:02:44.391798 25996 server_base.cc:1061] running on GCE node
I20260430 08:02:44.392517 25996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:02:44.393221 25996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:02:44.394430 25996 hybrid_clock.cc:648] HybridClock initialized: now 1777536164394416 us; error 33 us; skew 500 ppm
Apr 30 08:02:44 dist-test-slave-9bxp krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536164, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:02:44.398240 25996 init.cc:377] Logged in from keytab as kudu/127.13.94.129@KRBTEST.COM (short username kudu)
I20260430 08:02:44.399664 25996 webserver.cc:492] Webserver started at http://127.13.94.129:38275/ using document root <none> and password file <none>
I20260430 08:02:44.400409 25996 fs_manager.cc:362] Metadata directory not provided
I20260430 08:02:44.400470 25996 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:02:44.400720 25996 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 30 08:02:44 dist-test-slave-9bxp krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536164, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for HTTP/127.13.94.148@KRBTEST.COM
I20260430 08:02:44.429759 25996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/data/instance:
uuid: "d30363ca728b491cb78b7abc647d7e1a"
format_stamp: "Formatted at 2026-04-30 08:02:44 on dist-test-slave-9bxp"
server_key: "cb27bcbc47d7a278df1185a9261e2e3f"
server_key_iv: "66e59428ecb9c9f16d6b31e2c1d01546"
server_key_version: "kuduclusterkey@0"
I20260430 08:02:44.430377 25996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance:
uuid: "d30363ca728b491cb78b7abc647d7e1a"
format_stamp: "Formatted at 2026-04-30 08:02:44 on dist-test-slave-9bxp"
server_key: "cb27bcbc47d7a278df1185a9261e2e3f"
server_key_iv: "66e59428ecb9c9f16d6b31e2c1d01546"
server_key_version: "kuduclusterkey@0"
I20260430 08:02:44.438025 25996 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260430 08:02:44.460136 26013 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:02:44.469261 25996 fs_manager.cc:730] Time spent opening block manager: real 0.015s user 0.002s sys 0.001s
I20260430 08:02:44.469452 25996 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/wal
uuid: "d30363ca728b491cb78b7abc647d7e1a"
format_stamp: "Formatted at 2026-04-30 08:02:44 on dist-test-slave-9bxp"
server_key: "cb27bcbc47d7a278df1185a9261e2e3f"
server_key_iv: "66e59428ecb9c9f16d6b31e2c1d01546"
server_key_version: "kuduclusterkey@0"
I20260430 08:02:44.469606 25996 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:02:44.511195 25996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:02:44.515090 25996 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:02:44.515369 25996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:02:44.516067 25996 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:02:44.517356 25996 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:02:44.517480 25996 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:02:44.517608 25996 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:02:44.517697 25996 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:02:44.528993 25996 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.129:38821
I20260430 08:02:44.530380 25996 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/data/info.pb
I20260430 08:02:44.533305 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 25996
I20260430 08:02:44.533447 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-0/wal/instance
I20260430 08:02:44.533730 13690 external_mini_cluster.cc:1468] Setting key e10d96966dfd8852f53baf830c340415
I20260430 08:02:44.538491 26129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.129:38821 every 8 connection(s)
WARNING: no policy specified for kudu/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.94.130@KRBTEST.COM" created.
Apr 30 08:02:44 dist-test-slave-9bxp krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536164, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.129@KRBTEST.COM for kudu/127.13.94.190@KRBTEST.COM
I20260430 08:02:44.565590 26134 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:44.542387 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.13.94.129:46201 (local address 127.13.94.190:39047)
0430 08:02:44.547217 (+ 4830us) server_negotiation.cc:207] Beginning negotiation
0430 08:02:44.547222 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0430 08:02:44.547239 (+ 17us) server_negotiation.cc:408] Connection header received
0430 08:02:44.547286 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0430 08:02:44.547289 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0430 08:02:44.547344 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0430 08:02:44.547440 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0430 08:02:44.551423 (+ 3983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:02:44.552256 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0430 08:02:44.555213 (+ 2957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0430 08:02:44.555435 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:02:44.560979 (+ 5544us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0430 08:02:44.561013 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0430 08:02:44.561016 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0430 08:02:44.561052 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0430 08:02:44.563067 (+ 2015us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:02:44.563810 (+ 743us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:02:44.563814 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:02:44.563816 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:02:44.563888 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0430 08:02:44.564257 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0430 08:02:44.564261 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0430 08:02:44.564263 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0430 08:02:44.564444 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0430 08:02:44.564521 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0430 08:02:44.565266 (+ 745us) server_negotiation.cc:300] Negotiation successful
0430 08:02:44.565398 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4723,"thread_start_us":97,"threads_started":1}
I20260430 08:02:44.567379 26133 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:44.541392 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.129:46201)
0430 08:02:44.545580 (+ 4188us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:44.545634 (+ 54us) client_negotiation.cc:175] Beginning negotiation
0430 08:02:44.546698 (+ 1064us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0430 08:02:44.550045 (+ 3347us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0430 08:02:44.550060 (+ 15us) client_negotiation.cc:395] Received NEGOTIATE response from server
0430 08:02:44.550471 (+ 411us) client_negotiation.cc:190] Negotiated authn=SASL
0430 08:02:44.551240 (+ 769us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:02:44.551258 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:02:44.554246 (+ 2988us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0430 08:02:44.554251 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0430 08:02:44.554905 (+ 654us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0430 08:02:44.554917 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0430 08:02:44.555734 (+ 817us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0430 08:02:44.558822 (+ 3088us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0430 08:02:44.558869 (+ 47us) client_negotiation.cc:657] calling sasl_client_start()
0430 08:02:44.560686 (+ 1817us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0430 08:02:44.563229 (+ 2543us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:02:44.563236 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:02:44.563251 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:02:44.563672 (+ 421us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:02:44.564004 (+ 332us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0430 08:02:44.564007 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0430 08:02:44.564010 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0430 08:02:44.564151 (+ 141us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0430 08:02:44.564587 (+ 436us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0430 08:02:44.564593 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0430 08:02:44.564940 (+ 347us) client_negotiation.cc:770] Sending connection context
0430 08:02:44.566113 (+ 1173us) client_negotiation.cc:241] Negotiation successful
0430 08:02:44.566365 (+ 252us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3961,"thread_start_us":150,"threads_started":1}
I20260430 08:02:44.570159 26130 heartbeater.cc:344] Connected to a master server at 127.13.94.190:39047
I20260430 08:02:44.570438 26130 heartbeater.cc:461] Registering TS with master...
I20260430 08:02:44.571110 26130 heartbeater.cc:507] Master 127.13.94.190:39047 requested a full tablet report, sending...
I20260430 08:02:44.573035 25901 ts_manager.cc:194] Registered new tserver with Master: d30363ca728b491cb78b7abc647d7e1a (127.13.94.129:38821)
I20260430 08:02:44.574640 25901 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.94.129@KRBTEST.COM'} at 127.13.94.129:46201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.13.94.190 from keytab /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:878)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:745)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
WARNING: no policy specified for HTTP/127.13.94.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.94.130@KRBTEST.COM" created.
I20260430 08:02:44.614267 25988 server.cc:273] Received an EOF from the subprocess
F20260430 08:02:44.615207 25961 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7fea6b10edcd google::LogMessage::Fail() at ??:0
@ 0x7fea6b112b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fea6b10e7cc google::LogMessage::Flush() at ??:0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.94.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab.
I20260430 08:02:44.623653 13690 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
/tmp/dist-test-taskWlZwuK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.94.148:38483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.94.130:0
--local_ip_for_outbound_sockets=127.13.94.130
--webserver_interface=127.13.94.130
--webserver_port=0
--tserver_master_addrs=127.13.94.190:39047
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.94.148:38851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7fea6b10ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fea6cf27157 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fea6cf23f59 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fea6cf27e85 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x557cf2e30244 std::function<>::operator()() at ??:0
@ 0x7fea6c6137ea kudu::Thread::SuperviseThread() at ??:0
@ 0x7fea6cf576db start_thread at ??:0
@ 0x7fea6a2c561f clone at ??:0
W20260430 08:02:44.740216 26141 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260430 08:02:44.740442 26141 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260430 08:02:44.740499 26141 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260430 08:02:44.743916 26141 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260430 08:02:44.743985 26141 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260430 08:02:44.744102 26141 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.94.130
I20260430 08:02:44.748471 26141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.94.148:38851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.94.148:38483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.94.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.94.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.94.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.94.190:39047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.94.130
--log_dir=/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 30 Apr 2026 07:43:14 UTC on bdcb31816ec0
build id 11672
I20260430 08:02:44.749569 26141 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 08:02:44.750434 26141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 08:02:44.757092 26155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:02:44.757120 26158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 08:02:44.757092 26156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 08:02:44.757547 26141 server_base.cc:1061] running on GCE node
I20260430 08:02:44.757943 26141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260430 08:02:44.758498 26141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260430 08:02:44.759691 26141 hybrid_clock.cc:648] HybridClock initialized: now 1777536164759662 us; error 48 us; skew 500 ppm
Apr 30 08:02:44 dist-test-slave-9bxp krb5kdc[25145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536164, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260430 08:02:44.762749 26141 init.cc:377] Logged in from keytab as kudu/127.13.94.130@KRBTEST.COM (short username kudu)
I20260430 08:02:44.763883 26141 webserver.cc:492] Webserver started at http://127.13.94.130:40251/ using document root <none> and password file <none>
I20260430 08:02:44.764474 26141 fs_manager.cc:362] Metadata directory not provided
I20260430 08:02:44.764549 26141 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 08:02:44.764810 26141 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 30 08:02:44 dist-test-slave-9bxp krb5kdc[25145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777536164, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.94.130@KRBTEST.COM for HTTP/127.13.94.148@KRBTEST.COM
I20260430 08:02:44.781026 26141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/data/instance:
uuid: "5f1ab03ab3c549998216ecec474afb4a"
format_stamp: "Formatted at 2026-04-30 08:02:44 on dist-test-slave-9bxp"
server_key: "6fc34aaa61d0fcaad6e2f38a365f5185"
server_key_iv: "7ae85c4888d5e44649145881dbf1892f"
server_key_version: "kuduclusterkey@0"
I20260430 08:02:44.781814 26141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance:
uuid: "5f1ab03ab3c549998216ecec474afb4a"
format_stamp: "Formatted at 2026-04-30 08:02:44 on dist-test-slave-9bxp"
server_key: "6fc34aaa61d0fcaad6e2f38a365f5185"
server_key_iv: "7ae85c4888d5e44649145881dbf1892f"
server_key_version: "kuduclusterkey@0"
I20260430 08:02:44.786537 26141 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260430 08:02:44.802883 26165 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 08:02:44.804512 26141 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260430 08:02:44.804661 26141 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/wal
uuid: "5f1ab03ab3c549998216ecec474afb4a"
format_stamp: "Formatted at 2026-04-30 08:02:44 on dist-test-slave-9bxp"
server_key: "6fc34aaa61d0fcaad6e2f38a365f5185"
server_key_iv: "7ae85c4888d5e44649145881dbf1892f"
server_key_version: "kuduclusterkey@0"
I20260430 08:02:44.804847 26141 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 08:02:44.831694 26141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 08:02:44.835203 26141 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 08:02:44.835403 26141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 08:02:44.836068 26141 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 08:02:44.837193 26141 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 08:02:44.837258 26141 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 08:02:44.837312 26141 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 08:02:44.837342 26141 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 08:02:44.849712 26141 rpc_server.cc:307] RPC server started. Bound to: 127.13.94.130:37459
I20260430 08:02:44.849746 26278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.94.130:37459 every 8 connection(s)
I20260430 08:02:44.851035 26141 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/data/info.pb
I20260430 08:02:44.854423 13690 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWlZwuK/build/debug/bin/kudu as pid 26141
I20260430 08:02:44.854553 13690 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWlZwuK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777536011314967-13690-0/minicluster-data/ts-1/wal/instance
I20260430 08:02:44.854848 13690 external_mini_cluster.cc:1468] Setting key 45e960804bfad680fcc8d9a01c757baf
I20260430 08:02:44.855511 26281 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:44.854018 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.130:42669)
0430 08:02:44.854628 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:44.854739 (+ 111us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":155,"threads_started":1}
W20260430 08:02:44.856230 26279 heartbeater.cc:646] Failed to heartbeat to 127.13.94.190:39047 (0 consecutive failures): Network error: Failed to ping master at 127.13.94.190:39047: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
I20260430 08:02:44.856886 26281 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:44.856620 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.130:56961)
0430 08:02:44.856788 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:44.856828 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260430 08:02:44.857446 26281 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:44.857270 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.130:40949)
0430 08:02:44.857366 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:44.857401 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260430 08:02:44.857613 26279 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260430 08:02:45.577482 26130 heartbeater.cc:499] Master 127.13.94.190:39047 was elected leader, sending a full tablet report...
I20260430 08:02:45.578567 26283 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:45.578088 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.129:56919)
0430 08:02:45.578387 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:45.578456 (+ 69us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":113,"threads_started":1}
W20260430 08:02:45.578872 26130 heartbeater.cc:646] Failed to heartbeat to 127.13.94.190:39047 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
I20260430 08:02:45.859092 26284 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:45.858596 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.130:60191)
0430 08:02:45.858890 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:45.858952 (+ 62us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
I20260430 08:02:46.580061 26285 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:46.579536 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.129:46385)
0430 08:02:46.579839 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:46.579901 (+ 62us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260430 08:02:46.860606 26286 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:46.860125 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.130:42591)
0430 08:02:46.860429 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:46.860487 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260430 08:02:47.581472 26287 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:47.580996 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.129:42279)
0430 08:02:47.581316 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:47.581373 (+ 57us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":77,"threads_started":1}
I20260430 08:02:47.861944 26288 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:47.861497 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.130:50041)
0430 08:02:47.861787 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:47.861846 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260430 08:02:48.582798 26289 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:48.582347 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.129:36613)
0430 08:02:48.582635 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:48.582695 (+ 60us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
I20260430 08:02:48.863235 26290 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:48.862779 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.130:44657)
0430 08:02:48.863075 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:48.863134 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260430 08:02:49.584308 26291 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:49.583791 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.129:43807)
0430 08:02:49.584118 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:49.584205 (+ 87us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260430 08:02:49.864913 26292 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0430 08:02:49.864323 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.13.94.190:39047 (local address 127.13.94.130:40807)
0430 08:02:49.864670 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0430 08:02:49.864801 (+ 131us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.94.190:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":126,"threads_started":1}
F20260430 08:02:49.898258 13690 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1777536169 (unix time) try "date -d @1777536169" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000357a) received by PID 13690 (TID 0x7f23d30c8d80) from PID 13690; stack trace: ***
@ 0x7f23ded5a980 (unknown) at ??:0
@ 0x7f23dc3ace87 gsignal at ??:0
@ 0x7f23dc3ae7f1 abort at ??:0
@ 0x7f23dd619dcd google::LogMessage::Fail() at ??:0
@ 0x7f23dd61db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f23dd6197cc google::LogMessage::Flush() at ??:0
@ 0x7f23dd61af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f23e039c43b kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f23e0399156 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x5603105d37bf kudu::SecurityITest::StartCluster() at ??:0
@ 0x5603105bb622 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7f23de6250ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f23de619bdb testing::Test::Run() at ??:0
@ 0x7f23de619d9d testing::TestInfo::Run() at ??:0
@ 0x7f23de61a377 testing::TestSuite::Run() at ??:0
@ 0x7f23de61a77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f23de62560d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f23de619e63 testing::UnitTest::Run() at ??:0
@ 0x56031060bb0e RUN_ALL_TESTS() at ??:0
@ 0x56031060acfb main at ??:0
@ 0x7f23dc38fc87 __libc_start_main at ??:0
@ 0x5603105a2efa _start at ??:0
2026-04-30 08:02:50.091 UTC [25695] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 08:02:50.092 UTC [25846] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 08:02:50.093 UTC [25698] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 08:02:50.094 UTC [25697] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 08:02:50.096 UTC [25693] FATAL: terminating connection due to unexpected postmaster exit
2026-04-30 08:02:50.096 UTC [25694] FATAL: terminating connection due to unexpected postmaster exit